2024-03-21T21:03:11.227 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-21T21:03:11.228 DEBUG:teuthology.run:Teuthology command: teuthology --name phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} --owner scheduled_phlogistonjohn@teuthology --verbose --archive /home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118 -- /home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118/orig.config.yaml 2024-03-21T21:03:11.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T21:03:11.353 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118 branch: wip-phlogistonjohn-testing-2024-03-21-1131 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: jmulligan@redhat.com first_in_suite: false job_id: '7615118' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-phlogistonjohn-testing-2024-03-21-1131 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5d87ad5bd031c000b3eb3ea886a68e118ae09d38 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5d87ad5bd031c000b3eb3ea886a68e118ae09d38 workunit: branch: wip-phlogistonjohn-testing-2024-03-21-1131 sha1: 5d87ad5bd031c000b3eb3ea886a68e118ae09d38 owner: scheduled_phlogistonjohn@teuthology priority: 95 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 1248 sha1: 5d87ad5bd031c000b3eb3ea886a68e118ae09d38 sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-phlogistonjohn-testing-2024-03-21-1131 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5d87ad5bd031c000b3eb3ea886a68e118ae09d38/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5d87ad5bd031c000b3eb3ea886a68e118ae09d38 targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLT+w6gzfjApPcFVQPVNGJ+GM/Hovgy+qhTKc8HqxrjO92ZOc/GXZL9GTaOmmgnH3XPucBJusQF0Xx4Ay4DuFzM= smithi066.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEiwQxsdZiuPkLNNIeSatr9t2ofZNna2bdq/zpWeMK5Qe71r7U87/AkpS7nRo1oixXHMhrMWscMeHRk5t6o9gk= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-21_20:44:17 tube: smithi user: phlogistonjohn verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-21T21:03:11.353 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5d87ad5bd031c000b3eb3ea886a68e118ae09d38/qa; will attempt to use it 2024-03-21T21:03:11.354 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5d87ad5bd031c000b3eb3ea886a68e118ae09d38/qa/tasks 2024-03-21T21:03:11.354 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-21T21:03:11.355 INFO:teuthology.task.internal:Checking packages... 2024-03-21T21:03:11.384 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5d87ad5bd031c000b3eb3ea886a68e118ae09d38' 2024-03-21T21:03:11.384 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-21T21:03:11.384 INFO:teuthology.packaging:ref: None 2024-03-21T21:03:11.384 INFO:teuthology.packaging:tag: None 2024-03-21T21:03:11.384 INFO:teuthology.packaging:branch: wip-phlogistonjohn-testing-2024-03-21-1131 2024-03-21T21:03:11.384 INFO:teuthology.packaging:sha1: 5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:03:11.384 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-phlogistonjohn-testing-2024-03-21-1131 2024-03-21T21:03:11.610 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2286.g5d87ad5b 2024-03-21T21:03:11.612 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-21T21:03:11.619 INFO:teuthology.task.internal:no buildpackages task found 2024-03-21T21:03:11.620 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-21T21:03:11.692 INFO:teuthology.task.internal:Saving configuration 2024-03-21T21:03:11.709 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-21T21:03:11.716 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T21:03:11.741 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 20:56:36.632640', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLT+w6gzfjApPcFVQPVNGJ+GM/Hovgy+qhTKc8HqxrjO92ZOc/GXZL9GTaOmmgnH3XPucBJusQF0Xx4Ay4DuFzM='} 2024-03-21T21:03:11.765 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 20:56:36.631243', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEiwQxsdZiuPkLNNIeSatr9t2ofZNna2bdq/zpWeMK5Qe71r7U87/AkpS7nRo1oixXHMhrMWscMeHRk5t6o9gk='} 2024-03-21T21:03:11.765 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-21T21:03:11.771 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-03-21T21:03:11.771 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-21T21:03:11.771 INFO:teuthology.run_tasks:Running task console_log... 2024-03-21T21:03:11.859 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fda64b640d0>, signals=[15]) 2024-03-21T21:03:11.859 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-21T21:03:11.866 INFO:teuthology.task.internal:Opening connections... 2024-03-21T21:03:11.866 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T21:03:11.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:03:11.971 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2024-03-21T21:03:11.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:03:12.077 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-21T21:03:12.083 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-03-21T21:03:12.101 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-03-21T21:03:12.101 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-03-21T21:03:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-03-21T21:03:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-03-21T21:03:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-03-21T21:03:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-03-21T21:03:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-03-21T21:03:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-03-21T21:03:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T21:03:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-03-21T21:03:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-03-21T21:03:12.159 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T21:03:12.159 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-03-21T21:03:12.159 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T21:03:12.159 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T21:03:12.159 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T21:03:12.159 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-03-21T21:03:12.184 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2024-03-21T21:03:12.202 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2024-03-21T21:03:12.202 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2024-03-21T21:03:12.258 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Stream" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="9" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="9" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el9" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:LOGO="fedora-logo-icon" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://centos.org/" 2024-03-21T21:03:12.259 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T21:03:12.260 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T21:03:12.260 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T21:03:12.260 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2024-03-21T21:03:12.285 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-21T21:03:12.294 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-21T21:03:12.319 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-21T21:03:12.320 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-03-21T21:03:12.322 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2024-03-21T21:03:12.338 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-21T21:03:12.343 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-21T21:03:12.343 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-03-21T21:03:12.379 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2024-03-21T21:03:12.403 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-21T21:03:12.459 INFO:teuthology.run_tasks:Running task kernel... 2024-03-21T21:03:12.472 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-21T21:03:12.473 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-21T21:03:12.473 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T21:03:12.473 DEBUG:teuthology.orchestra.run.smithi066:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T21:03:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T21:03:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T21:03:12.491 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-21T21:03:12.492 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-03-21T21:03:12.549 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-361.el9.x86_64 2024-03-21T21:03:12.549 INFO:teuthology.task.kernel:Running kernel on smithi066: 5.14.0-361.el9.x86_64 2024-03-21T21:03:12.549 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum install -y kernel 2024-03-21T21:03:12.550 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-361.el9.x86_64 2024-03-21T21:03:12.550 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-361.el9.x86_64 2024-03-21T21:03:12.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-03-21T21:03:14.641 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-21T21:03:14.686 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-21T21:03:16.166 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-21T21:03:16.187 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-21T21:03:18.352 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-03-21T21:03:18.365 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-21T21:03:20.516 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-21T21:03:24.544 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 3.7 MB/s | 19 MB 00:05 2024-03-21T21:03:25.067 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.1 MB 00:00 2024-03-21T21:03:26.459 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 16 kB 00:00 2024-03-21T21:03:46.995 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-21T21:03:52.075 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-03-21T21:03:52.551 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 257 kB/s | 6.1 MB 00:24 2024-03-21T21:03:54.227 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-21T21:03:54.482 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T21:03:54.483 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T21:03:54.579 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-21T21:03:54.591 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-21T21:03:54.591 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-21T21:03:54.591 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-21T21:03:54.591 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-21T21:03:54.591 INFO:teuthology.orchestra.run.smithi066.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T21:03:54.592 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-21T21:03:54.592 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T21:03:54.592 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T21:03:54.592 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T21:03:54.592 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:03:54.592 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-21T21:03:54.592 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-21T21:03:54.592 INFO:teuthology.orchestra.run.smithi066.stdout:Install 4 Packages 2024-03-21T21:03:54.592 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:03:54.593 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 76 M 2024-03-21T21:03:54.593 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 123 M 2024-03-21T21:03:54.593 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-21T21:03:55.054 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-21T21:03:55.535 INFO:teuthology.orchestra.run.smithi066.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 67 kB/s | 44 kB 00:00 2024-03-21T21:04:00.162 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-21T21:04:02.352 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T21:04:02.352 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T21:04:02.461 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-21T21:04:02.473 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-21T21:04:02.473 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-03-21T21:04:02.473 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-21T21:04:02.473 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-21T21:04:02.473 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 76 M 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 123 M 2024-03-21T21:04:02.474 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-21T21:04:02.913 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 142 kB/s | 44 kB 00:00 2024-03-21T21:04:03.530 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-21T21:04:04.222 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 21 MB/s | 33 MB 00:01 2024-03-21T21:04:04.472 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 18 MB/s | 27 MB 00:01 2024-03-21T21:04:04.476 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-21T21:04:04.476 INFO:teuthology.orchestra.run.smithi028.stdout:Total 38 MB/s | 76 MB 00:02 2024-03-21T21:04:05.169 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-21T21:04:05.570 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-21T21:04:05.571 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-21T21:04:06.630 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-21T21:04:06.631 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-21T21:04:09.238 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-21T21:04:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T21:04:09.981 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T21:04:12.843 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T21:04:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T21:04:14.125 INFO:teuthology.orchestra.run.smithi066.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 793 kB/s | 15 MB 00:19 2024-03-21T21:04:18.052 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T21:04:18.092 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:04:23.193 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:04:31.990 INFO:teuthology.orchestra.run.smithi066.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 771 kB/s | 27 MB 00:36 2024-03-21T21:04:37.556 INFO:teuthology.orchestra.run.smithi066.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 802 kB/s | 33 MB 00:42 2024-03-21T21:04:37.557 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-21T21:04:37.557 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.8 MB/s | 76 MB 00:42 2024-03-21T21:04:38.244 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-21T21:04:38.660 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-21T21:04:38.660 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-21T21:04:39.713 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-21T21:04:39.713 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-21T21:04:42.349 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-21T21:04:42.971 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T21:04:43.154 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T21:04:46.052 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T21:04:46.183 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T21:04:51.287 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T21:04:51.332 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:04:54.174 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:04:54.174 INFO:teuthology.orchestra.run.smithi028.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T21:04:54.174 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:04:54.219 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:04:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:04:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T21:04:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T21:04:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T21:04:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:04:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:04:55.040 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-21T21:04:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T21:04:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T21:04:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T21:04:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T21:04:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:04:55.041 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-21T21:04:55.363 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T21:04:55.469 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T21:04:55.470 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T21:04:55.470 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T21:04:55.470 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T21:04:55.470 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T21:04:55.470 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-21T21:04:55.470 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T21:04:55.470 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T21:04:55.553 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T21:04:55.553 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T21:04:55.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-03-21T21:04:55.592 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-03-21T21:04:55.592 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-03-21T21:04:55.592 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-03-21T21:04:55.592 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-03-21T21:04:55.592 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-03-21T21:04:55.592 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T21:04:55.592 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-03-21T21:04:55.592 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T21:04:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-03-21T21:04:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:04:55.594 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T21:04:55.594 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T21:04:55.595 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-430.el9.x86_64 2024-03-21T21:04:55.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T21:04:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-03-21T21:04:56.469 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:04:57.775 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T21:04:57.820 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-03-21T21:04:57.822 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-03-21T21:04:57.838 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.Mx8KXkq8zm 2024-03-21T21:04:57.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Mx8KXkq8zm 2024-03-21T21:04:57.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.Mx8KXkq8zm 2024-03-21T21:04:58.053 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.Mx8KXkq8zm is 6KB 2024-03-21T21:04:58.101 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.Mx8KXkq8zm 2024-03-21T21:04:58.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T21:04:58.184 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T21:04:58.185 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T21:04:58.185 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T21:04:58.185 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T21:04:58.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T21:04:58.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-03-21T21:05:27.385 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:05:27.385 INFO:teuthology.orchestra.run.smithi066.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T21:05:27.385 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:05:27.430 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:05:27.809 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:05:27.810 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T21:05:27.810 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T21:05:27.810 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T21:05:28.210 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T21:05:28.210 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:05:28.211 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-21T21:05:28.211 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T21:05:28.211 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T21:05:28.211 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T21:05:28.211 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T21:05:28.211 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:05:28.211 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-21T21:05:28.329 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-21T21:05:28.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T21:05:28.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:05:28.528 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T21:05:28.648 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T21:05:28.648 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T21:05:28.648 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T21:05:28.648 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T21:05:28.648 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T21:05:28.648 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-21T21:05:28.648 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T21:05:28.649 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T21:05:28.731 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T21:05:28.731 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T21:05:28.732 DEBUG:teuthology.orchestra.run.smithi066:> sudo rpm -qi grub2-tools 2024-03-21T21:05:28.769 INFO:teuthology.orchestra.run.smithi066.stdout:Name : grub2-tools 2024-03-21T21:05:28.769 INFO:teuthology.orchestra.run.smithi066.stdout:Epoch : 1 2024-03-21T21:05:28.769 INFO:teuthology.orchestra.run.smithi066.stdout:Version : 2.06 2024-03-21T21:05:28.769 INFO:teuthology.orchestra.run.smithi066.stdout:Release : 68.el9 2024-03-21T21:05:28.769 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: x86_64 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Group : Unspecified 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Size : 8264254 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:License : GPLv3+ 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Packager : builder@centos.org 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Vendor : CentOS 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Summary : Support tools for GRUB. 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:Description : 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:05:28.770 INFO:teuthology.orchestra.run.smithi066.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T21:05:28.771 INFO:teuthology.orchestra.run.smithi066.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T21:05:28.771 INFO:teuthology.orchestra.run.smithi066.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T21:05:28.771 INFO:teuthology.orchestra.run.smithi066.stdout:hardware devices. 2024-03-21T21:05:28.771 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:05:28.771 INFO:teuthology.orchestra.run.smithi066.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T21:05:28.772 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T21:05:28.772 INFO:teuthology.task.kernel:Updating grub on smithi066 to boot 5.14.0-430.el9.x86_64 2024-03-21T21:05:28.772 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T21:05:29.395 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2024-03-21T21:05:30.690 INFO:teuthology.orchestra.run.smithi066.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T21:05:30.721 INFO:teuthology.orchestra.run.smithi066.stderr:done 2024-03-21T21:05:30.723 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-03-21T21:05:30.738 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.O0a0mRnMKO 2024-03-21T21:05:30.739 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.O0a0mRnMKO 2024-03-21T21:05:30.810 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.O0a0mRnMKO 2024-03-21T21:05:30.954 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.O0a0mRnMKO is 6KB 2024-03-21T21:05:31.003 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.O0a0mRnMKO 2024-03-21T21:05:31.018 DEBUG:teuthology.orchestra.run.smithi066:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T21:05:31.094 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T21:05:31.094 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T21:05:31.094 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T21:05:31.095 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T21:05:31.096 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T21:05:31.273 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2024-03-21T21:06:01.302 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-21T21:06:01.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-21T21:06:01.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:06:28.359 DEBUG:teuthology.orchestra.remote:timed out 2024-03-21T21:06:35.938 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-21T21:06:37.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T21:06:37.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:06:37.667 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-03-21T21:06:38.022 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T21:06:38.022 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-21T21:06:38.022 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-03-21T21:06:38.037 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-430.el9.x86_64 2024-03-21T21:06:38.037 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T21:06:38.037 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T21:06:38.037 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T21:06:39.039 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-21T21:06:39.039 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T21:06:39.123 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-03-21T21:06:39.153 DEBUG:teuthology.parallel:result is None 2024-03-21T21:06:44.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-21T21:06:44.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:06:47.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-21T21:06:59.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-21T21:07:00.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:07:03.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-21T21:07:18.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-21T21:07:18.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:07:21.153 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-21T21:07:39.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-21T21:07:39.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:08:11.052 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-21T21:08:11.433 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-21T21:08:11.434 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-21T21:08:11.434 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-21T21:08:11.449 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-430.el9.x86_64 2024-03-21T21:08:11.450 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T21:08:11.450 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T21:08:11.450 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T21:08:12.451 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-21T21:08:12.451 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T21:08:12.537 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2024-03-21T21:08:12.575 DEBUG:teuthology.parallel:result is None 2024-03-21T21:08:12.575 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-21T21:08:12.583 INFO:teuthology.task.internal:Creating test directory... 2024-03-21T21:08:12.583 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T21:08:12.586 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T21:08:12.604 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-21T21:08:12.611 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-21T21:08:12.710 INFO:teuthology.task.internal:Creating archive directory... 2024-03-21T21:08:12.710 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T21:08:12.713 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T21:08:12.747 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-21T21:08:12.759 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-21T21:08:12.759 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T21:08:12.778 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T21:08:12.820 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T21:08:12.820 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T21:08:12.834 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T21:08:12.837 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T21:08:12.839 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-21T21:08:12.846 INFO:teuthology.task.internal:Configuring sudo... 2024-03-21T21:08:12.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T21:08:12.878 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T21:08:12.912 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-21T21:08:12.921 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-21T21:08:12.921 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T21:08:12.946 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T21:08:12.968 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T21:08:13.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T21:08:13.092 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T21:08:13.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T21:08:13.217 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:08:13.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T21:08:13.284 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T21:08:13.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T21:08:13.382 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T21:08:13.465 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T21:08:13.513 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-21T21:08:13.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T21:08:13.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-03-21T21:08:13.587 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2024-03-21T21:08:13.640 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T21:08:13.686 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T21:08:14.025 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-21T21:08:14.034 INFO:teuthology.task.internal:Starting timer... 2024-03-21T21:08:14.034 INFO:teuthology.run_tasks:Running task pcp... 2024-03-21T21:08:14.072 INFO:teuthology.run_tasks:Running task selinux... 2024-03-21T21:08:14.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-03-21T21:08:14.181 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-03-21T21:08:14.183 DEBUG:teuthology.orchestra.run.smithi066:> sudo service auditd rotate 2024-03-21T21:08:14.229 INFO:teuthology.orchestra.run.smithi066.stdout:Rotating logs: 2024-03-21T21:08:14.231 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-21T21:08:14.231 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-03-21T21:08:14.252 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-03-21T21:08:14.252 DEBUG:teuthology.orchestra.run.smithi066:> /usr/sbin/getenforce 2024-03-21T21:08:14.294 INFO:teuthology.orchestra.run.smithi066.stdout:Permissive 2024-03-21T21:08:14.294 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi066.front.sepia.ceph.com': 'permissive'} 2024-03-21T21:08:14.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-21T21:08:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T21:08:14.322 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-21T21:08:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T21:08:14.365 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-21T21:08:14.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-03-21T21:08:14.396 DEBUG:teuthology.orchestra.run.smithi066:> sudo /usr/sbin/setenforce permissive 2024-03-21T21:08:14.447 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-21T21:08:14.457 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-21T21:08:14.474 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-21T21:08:14.475 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi066.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-21T21:12:41.119 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi066.front.sepia.ceph.com')] 2024-03-21T21:12:41.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T21:12:41.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:12:41.217 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-03-21T21:12:41.298 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T21:12:41.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-21T21:12:41.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T21:12:41.401 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-21T21:12:41.483 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-21T21:12:41.483 INFO:teuthology.run_tasks:Running task clock... 2024-03-21T21:12:41.493 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-21T21:12:41.493 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T21:12:41.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T21:12:41.496 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T21:12:41.496 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T21:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T21:12:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T21:12:41.561 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T21:12:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-03-21T21:12:41.581 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T21:12:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-03-21T21:12:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T21:12:41.618 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: ntpd: command not found 2024-03-21T21:12:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T21:12:41.635 INFO:teuthology.orchestra.run.smithi066.stdout:506 Cannot talk to daemon 2024-03-21T21:12:41.655 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T21:12:41.675 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T21:12:41.676 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-03-21T21:12:41.679 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T21:12:41.679 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-03-21T21:12:41.679 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:12:41.679 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:12:41.679 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:12:41.679 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:12:41.730 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-21T21:12:41.733 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T21:12:41.734 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-21T21:12:41.734 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:12:41.734 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:12:41.734 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:12:41.734 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:12:41.735 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-21T21:12:41.840 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5d87ad5bd031c000b3eb3ea886a68e118ae09d38'} 2024-03-21T21:12:41.840 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-21T21:12:41.841 INFO:tasks.cephadm:Cluster fsid is c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:12:41.841 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-21T21:12:41.841 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.c': '[v2:172.21.15.28:3301,v1:172.21.15.28:6790]', 'mon.b': '172.21.15.66'} 2024-03-21T21:12:41.841 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-03-21T21:12:41.841 INFO:tasks.cephadm:First mgr is y 2024-03-21T21:12:41.841 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-21T21:12:41.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-03-21T21:12:41.869 DEBUG:teuthology.orchestra.run.smithi066:> sudo hostname $(hostname -s) 2024-03-21T21:12:41.902 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-21T21:12:41.903 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T21:12:42.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 21 21:12 /home/ubuntu/cephtest/cephadm 2024-03-21T21:12:42.182 DEBUG:teuthology.orchestra.run.smithi066:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T21:12:42.402 INFO:teuthology.orchestra.run.smithi066.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 21 21:12 /home/ubuntu/cephtest/cephadm 2024-03-21T21:12:42.402 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T21:12:42.418 DEBUG:teuthology.orchestra.run.smithi066:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T21:12:42.446 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-21T21:12:42.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-21T21:12:42.460 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-21T21:12:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-21T21:12:43.485 INFO:teuthology.orchestra.run.smithi066.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-21T21:13:05.148 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:13:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-21T21:13:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-21T21:13:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-03-21T21:13:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-21T21:13:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-21T21:13:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-21T21:13:05.149 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:13:05.207 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-21T21:13:05.207 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-21T21:13:05.208 INFO:teuthology.orchestra.run.smithi066.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-21T21:13:05.208 INFO:teuthology.orchestra.run.smithi066.stdout: "repo_digests": [ 2024-03-21T21:13:05.208 INFO:teuthology.orchestra.run.smithi066.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-21T21:13:05.208 INFO:teuthology.orchestra.run.smithi066.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-21T21:13:05.208 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-21T21:13:05.208 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-21T21:13:05.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-03-21T21:13:05.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph 2024-03-21T21:13:05.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-03-21T21:13:05.323 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 777 /etc/ceph 2024-03-21T21:13:05.361 INFO:tasks.cephadm:Writing seed config... 2024-03-21T21:13:05.362 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-21T21:13:05.362 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-21T21:13:05.363 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-21T21:13:05.363 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-21T21:13:05.363 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-21T21:13:05.363 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-21T21:13:05.363 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-21T21:13:05.363 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-21T21:13:05.364 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-21T21:13:05.365 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:13:05.365 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-21T21:13:05.380 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c10cc158-e7c7-11ee-95cd-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-21T21:13:05.380 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.a.service 2024-03-21T21:13:05.422 DEBUG:teuthology.orchestra.run.smithi028:mgr.y> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.y.service 2024-03-21T21:13:05.464 INFO:tasks.cephadm:Bootstrapping... 2024-03-21T21:13:05.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-21T21:13:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:-------------------------------------------------------------------------------- 2024-03-21T21:13:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'c10cc158-e7c7-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-03-21T21:13:05.686 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: 4.9.3-dev 2024-03-21T21:13:05.688 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying podman|docker is present... 2024-03-21T21:13:05.710 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: 4.9.3-dev 2024-03-21T21:13:05.712 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying lvm2 is present... 2024-03-21T21:13:05.712 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying time synchronization is in place... 2024-03-21T21:13:05.719 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T21:13:05.725 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-03-21T21:13:05.731 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: enabled 2024-03-21T21:13:05.737 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: active 2024-03-21T21:13:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:Unit chronyd.service is enabled and running 2024-03-21T21:13:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:Repeating the final host check... 2024-03-21T21:13:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: 4.9.3-dev 2024-03-21T21:13:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-21T21:13:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl is present 2024-03-21T21:13:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:lvcreate is present 2024-03-21T21:13:05.766 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T21:13:05.772 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-03-21T21:13:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: enabled 2024-03-21T21:13:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: active 2024-03-21T21:13:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:Unit chronyd.service is enabled and running 2024-03-21T21:13:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:Host looks OK 2024-03-21T21:13:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:Cluster fsid: c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:Acquiring lock 139728164011936 on /run/cephadm/c10cc158-e7c7-11ee-95cd-87774f69a715.lock 2024-03-21T21:13:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:Lock 139728164011936 acquired on /run/cephadm/c10cc158-e7c7-11ee-95cd-87774f69a715.lock 2024-03-21T21:13:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying IP 172.21.15.28 port 3300 ... 2024-03-21T21:13:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying IP 172.21.15.28 port 6789 ... 2024-03-21T21:13:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:Base mon IP is 172.21.15.28, final addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-03-21T21:13:05.788 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-03-21T21:13:05.788 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-03-21T21:13:05.791 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-21T21:13:05.791 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-21T21:13:05.791 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-21T21:13:05.793 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-21T21:13:05.793 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-21T21:13:05.793 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-21T21:13:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-21T21:13:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-03-21T21:13:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-21T21:13:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-03-21T21:13:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-21T21:13:05.795 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-21T21:13:05.900 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-21T21:13:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Getting image source signatures 2024-03-21T21:13:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-21T21:13:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-21T21:13:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-21T21:13:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-21T21:13:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-21T21:13:06.250 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-21T21:13:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Writing manifest to image destination 2024-03-21T21:13:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-21T21:13:06.914 INFO:teuthology.orchestra.run.smithi028.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-21T21:13:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-21T21:13:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:Extracting ceph user uid/gid from container image... 2024-03-21T21:13:07.827 INFO:teuthology.orchestra.run.smithi028.stderr:stat: 167 167 2024-03-21T21:13:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:Creating initial keys... 2024-03-21T21:13:08.782 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph-authtool: AQDkovxlEv10KhAAWBAgQ6jNzpDElHZwYQZehQ== 2024-03-21T21:13:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph-authtool: AQDlovxlhb86KRAAebOS92kp9umehx/EB/V2SA== 2024-03-21T21:13:10.792 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph-authtool: AQDmovxlMPcQKxAALKi1iC7SRh9MTZg7DoGk/w== 2024-03-21T21:13:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:Creating initial monmap... 2024-03-21T21:13:11.698 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T21:13:11.698 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-21T21:13:11.698 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:11.698 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T21:13:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T21:13:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:setting min_mon_release = octopus 2024-03-21T21:13:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: set fsid to c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T21:13:12.254 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:13:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:Creating mon... 2024-03-21T21:13:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:create mon.a on 2024-03-21T21:13:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-21T21:13:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715.target. 2024-03-21T21:13:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715.target. 2024-03-21T21:13:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Failed to reset failed state of unit ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-21T21:13:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715.target.wants/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@.service. 2024-03-21T21:13:14.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:14 smithi028 podman[24432]: 2024-03-21 21:13:14.186814257 +0000 UTC m=+0.019018713 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-21T21:13:15.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:15 smithi028 podman[24432]: 2024-03-21 21:13:15.1400992 +0000 UTC m=+0.972303649 container create ba0a1747ac51fdd3a2b65980e94e8fc62bb12fa2260d9e9defadf481b6c21c74 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, release=754, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , ceph=True, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, version=8, RELEASE=HEAD, GIT_CLEAN=True, architecture=x86_64, vcs-type=git, io.openshift.tags=base centos centos-stream) 2024-03-21T21:13:17.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:16 smithi028 podman[24432]: 2024-03-21 21:13:16.753968387 +0000 UTC m=+2.586172849 container init ba0a1747ac51fdd3a2b65980e94e8fc62bb12fa2260d9e9defadf481b6c21c74 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-type=git, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, name=centos-stream, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-21T21:13:17.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:16 smithi028 podman[24432]: 2024-03-21 21:13:16.757721818 +0000 UTC m=+2.589926267 container start ba0a1747ac51fdd3a2b65980e94e8fc62bb12fa2260d9e9defadf481b6c21c74 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, version=8, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, release=754, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, ceph=True, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD) 2024-03-21T21:13:17.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:16 smithi028 ceph-mon[24450]: mkfs c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:17.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:16 smithi028 ceph-mon[24450]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T21:13:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: disabled 2024-03-21T21:13:17.568 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-03-21T21:13:17.569 INFO:teuthology.orchestra.run.smithi028.stderr:firewalld.service is not enabled 2024-03-21T21:13:17.569 INFO:teuthology.orchestra.run.smithi028.stderr:Not possible to enable service . firewalld.service is not available 2024-03-21T21:13:17.570 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mon to start... 2024-03-21T21:13:17.570 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mon... 2024-03-21T21:13:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:17 smithi028 bash[24432]: ba0a1747ac51fdd3a2b65980e94e8fc62bb12fa2260d9e9defadf481b6c21c74 2024-03-21T21:13:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:17 smithi028 systemd[1]: Started Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:13:18.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:17 smithi028 ceph-mon[24450]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T21:13:18.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:17 smithi028 ceph-mon[24450]: monmap e1: 1 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0]} 2024-03-21T21:13:18.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:17 smithi028 ceph-mon[24450]: fsmap 2024-03-21T21:13:18.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:17 smithi028 ceph-mon[24450]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T21:13:18.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:17 smithi028 ceph-mon[24450]: mgrmap e1: no daemons active 2024-03-21T21:13:20.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:20 smithi028 ceph-mon[24450]: from='client.? 172.21.15.28:0/2110346254' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:13:21.115 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: cluster: 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: id: c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: services: 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 3s) 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: data: 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: pgs: 2024-03-21T21:13:21.116 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:mon is available 2024-03-21T21:13:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:Assimilating anything we can from ceph.conf... 2024-03-21T21:13:24.973 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:24.973 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [global] 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: fsid = c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [mgr] 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [osd] 2024-03-21T21:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-21T21:13:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-21T21:13:25.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:25 smithi028 ceph-mon[24450]: from='client.? 172.21.15.28:0/2713624228' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T21:13:25.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:25 smithi028 ceph-mon[24450]: from='client.? 172.21.15.28:0/2713624228' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T21:13:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:Generating new minimal ceph.conf... 2024-03-21T21:13:26.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:26 smithi028 ceph-mon[24450]: from='client.? 172.21.15.28:0/1093035865' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:13:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:Restarting the monitor... 2024-03-21T21:13:26.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:26 smithi028 systemd[1]: Stopping Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:13:26.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:26 smithi028 bash[24679]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon.a" found: no such container 2024-03-21T21:13:27.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:26 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a[24446]: 2024-03-21T21:13:26.850+0000 7f0a2bbd6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:13:27.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:26 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a[24446]: 2024-03-21T21:13:26.850+0000 7f0a2bbd6700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-21T21:13:27.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 podman[24692]: 2024-03-21 21:13:27.045462616 +0000 UTC m=+0.305293253 container died ba0a1747ac51fdd3a2b65980e94e8fc62bb12fa2260d9e9defadf481b6c21c74 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_CLEAN=True, version=8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, name=centos-stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-21T21:13:27.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 podman[24692]: 2024-03-21 21:13:27.416779235 +0000 UTC m=+0.676609865 container cleanup ba0a1747ac51fdd3a2b65980e94e8fc62bb12fa2260d9e9defadf481b6c21c74 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, version=8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.openshift.expose-services=, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-21T21:13:27.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 bash[24692]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a 2024-03-21T21:13:27.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 bash[24717]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon.a" found: no such container 2024-03-21T21:13:27.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 podman[24706]: 2024-03-21 21:13:27.692521277 +0000 UTC m=+0.639020564 container remove ba0a1747ac51fdd3a2b65980e94e8fc62bb12fa2260d9e9defadf481b6c21c74 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.openshift.expose-services=, ceph=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_CLEAN=True, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, release=754, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-type=git, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-21T21:13:27.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-21T21:13:27.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 systemd[1]: Stopped Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:13:27.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 systemd[1]: Starting Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:13:27.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 podman[24787]: 2024-03-21 21:13:27.918338089 +0000 UTC m=+0.107070699 container create 355fc6af68b078ea4037f61d19a370f967492c1558bd826df4b2525cb699cbdf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, ceph=True, io.openshift.expose-services=, release=754, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, version=8, io.k8s.display-name=CentOS Stream 8) 2024-03-21T21:13:27.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:27 smithi028 podman[24787]: 2024-03-21 21:13:27.829901137 +0000 UTC m=+0.018633734 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-21T21:13:28.168 INFO:teuthology.orchestra.run.smithi028.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-21T21:13:28.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 podman[24787]: 2024-03-21 21:13:28.095367378 +0000 UTC m=+0.284099956 container init 355fc6af68b078ea4037f61d19a370f967492c1558bd826df4b2525cb699cbdf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, release=754, io.openshift.expose-services=, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., ceph=True, name=centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0) 2024-03-21T21:13:28.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 podman[24787]: 2024-03-21 21:13:28.098152247 +0000 UTC m=+0.286884819 container start 355fc6af68b078ea4037f61d19a370f967492c1558bd826df4b2525cb699cbdf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, ceph=True, name=centos-stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-03-21T21:13:28.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T21:13:28.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-21T21:13:28.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: pidfile_write: ignore empty --pid-file 2024-03-21T21:13:28.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: load: jerasure load: lrc 2024-03-21T21:13:28.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: RocksDB version: 6.15.5 2024-03-21T21:13:28.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-21T21:13:28.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Compile date Apr 18 2022 2024-03-21T21:13:28.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: DB SUMMARY 2024-03-21T21:13:28.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: DB Session ID: HWM3S8D3UFHCLEBXP67U 2024-03-21T21:13:28.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: CURRENT file: CURRENT 2024-03-21T21:13:28.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T21:13:28.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-21T21:13:28.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-21T21:13:28.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83934 ; 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.error_if_exists: 0 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.create_if_missing: 0 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.paranoid_checks: 1 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.env: 0x55b5aaec6860 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.fs: Posix File System 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.info_log: 0x55b5abe0bdc0 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.statistics: (nil) 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.use_fsync: 0 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_log_file_size: 0 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T21:13:28.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.allow_fallocate: 1 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.use_direct_reads: 0 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.db_log_dir: 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T21:13:28.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.write_buffer_manager: 0x55b5abefc2a0 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.unordered_write: 0 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T21:13:28.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.row_cache: None 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.wal_filter: None 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.preserve_deletes: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.two_write_queues: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.atomic_flush: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.log_readahead_size: 0 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T21:13:28.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_background_jobs: 2 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_background_compactions: -1 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_subcompactions: 1 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T21:13:28.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T21:13:28.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T21:13:28.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_open_files: -1 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_background_flushes: -1 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Compression algorithms supported: 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: kZSTD supported: 0 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: kXpressCompression supported: 0 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: kLZ4Compression supported: 1 2024-03-21T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: kBZip2Compression supported: 0 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: kZlibCompression supported: 1 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: kSnappyCompression supported: 1 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.merge_operator: 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_filter: None 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b5abdd7d00) 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-03-21T21:13:28.300 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: hash_index_allow_collision: 1 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 1 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x55b5abe42170 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-03-21T21:13:28.301 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-03-21T21:13:28.302 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 4 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compression: NoCompression 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.num_levels: 7 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T21:13:28.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T21:13:28.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T21:13:28.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.arena_block_size: 4194304 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T21:13:28.306 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.table_properties_collectors: 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.inplace_update_support: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.bloom_locality: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.max_successive_merges: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.ttl: 2592000 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T21:13:28.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.enable_blob_files: false 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.min_blob_size: 0 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711055608123468, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711055608124197, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79887, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78131, "index_size": 244, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10089, "raw_average_key_size": 46, "raw_value_size": 72368, "raw_average_value_size": 336, "num_data_blocks": 11, "num_entries": 215, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711055608, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "505b0e86-ff51-4d09-8599-6a936d9016df", "db_session_id": "HWM3S8D3UFHCLEBXP67U"}} 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711055608124521, "job": 1, "event": "recovery_finished"} 2024-03-21T21:13:28.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55b5abe28a80 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: DB pointer 0x55b5abe38000 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:13:28.309 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 79.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 79.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T21:13:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 26.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 26.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 79.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 79.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2024-03-21T21:13:28.311 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.0 0.00 0.00 1 0.000 0 0 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 25.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:13:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mon.a@-1(???) e1 preinit fsid c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mon.a@-1(???).mds e1 new map 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mon.a@-1(???).mds e1 print_map 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T21:13:28.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta expand map: {default=false} 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta from 'false' to 'false' 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta expanded map: {default=false} 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta expand map: {default=info} 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta from 'info' to 'info' 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta expanded map: {default=info} 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta expand map: {default=daemon} 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta from 'daemon' to 'daemon' 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta expanded map: {default=daemon} 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta expand map: {default=debug} 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta from 'debug' to 'debug' 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: expand_channel_meta expanded map: {default=debug} 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 bash[24787]: 355fc6af68b078ea4037f61d19a370f967492c1558bd826df4b2525cb699cbdf 2024-03-21T21:13:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 systemd[1]: Started Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:13:28.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T21:13:28.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: monmap e1: 1 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0]} 2024-03-21T21:13:28.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: fsmap 2024-03-21T21:13:28.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T21:13:28.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:28 smithi028 ceph-mon[24806]: mgrmap e1: no daemons active 2024-03-21T21:13:30.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:29 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3612489001' entity='client.admin' 2024-03-21T21:13:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-21T21:13:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-21T21:13:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:Creating mgr... 2024-03-21T21:13:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying port 9283 ... 2024-03-21T21:13:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Failed to reset failed state of unit ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-21T21:13:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715.target.wants/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@.service. 2024-03-21T21:13:32.333 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:32 smithi028 systemd[1]: Starting Ceph mgr.y for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:13:32.627 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:32 smithi028 podman[25023]: 2024-03-21 21:13:32.345951917 +0000 UTC m=+0.105740524 container create 602bce1ea97f017f8c18b99843a57d9c364f5a86b84a9bc195236bb02ff3f824 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, io.openshift.tags=base centos centos-stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, version=8, distribution-scope=public, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-21T21:13:32.627 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:32 smithi028 podman[25023]: 2024-03-21 21:13:32.258213908 +0000 UTC m=+0.018002546 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-21T21:13:32.627 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:32 smithi028 podman[25023]: 2024-03-21 21:13:32.556749943 +0000 UTC m=+0.316538567 container init 602bce1ea97f017f8c18b99843a57d9c364f5a86b84a9bc195236bb02ff3f824 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vcs-type=git, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8, architecture=x86_64, name=centos-stream, distribution-scope=public, version=8, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc.) 2024-03-21T21:13:32.627 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:32 smithi028 podman[25023]: 2024-03-21 21:13:32.559275222 +0000 UTC m=+0.319063832 container start 602bce1ea97f017f8c18b99843a57d9c364f5a86b84a9bc195236bb02ff3f824 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, RELEASE=HEAD) 2024-03-21T21:13:32.627 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:32 smithi028 bash[25023]: 602bce1ea97f017f8c18b99843a57d9c364f5a86b84a9bc195236bb02ff3f824 2024-03-21T21:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: disabled 2024-03-21T21:13:32.646 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-03-21T21:13:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:firewalld.service is not enabled 2024-03-21T21:13:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:Not possible to enable service . firewalld.service is not available 2024-03-21T21:13:32.654 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: disabled 2024-03-21T21:13:32.662 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-03-21T21:13:32.662 INFO:teuthology.orchestra.run.smithi028.stderr:firewalld.service is not enabled 2024-03-21T21:13:32.663 INFO:teuthology.orchestra.run.smithi028.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-21T21:13:32.663 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mgr to start... 2024-03-21T21:13:32.663 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mgr... 2024-03-21T21:13:32.921 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:32 smithi028 systemd[1]: Started Ceph mgr.y for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:13:33.104 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:33.104 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-03-21T21:13:33.104 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsid": "c10cc158-e7c7-11ee-95cd-87774f69a715", 2024-03-21T21:13:33.104 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "health": { 2024-03-21T21:13:33.104 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-21T21:13:33.104 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "checks": {}, 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mutes": [] 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum": [ 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 0 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "a" 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "monmap": { 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osdmap": { 2024-03-21T21:13:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-21T21:13:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-21T21:13:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-21T21:13:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-21T21:13:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-21T21:13:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-21T21:13:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgmap": { 2024-03-21T21:13:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsmap": { 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-21T21:13:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": false, 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modules": [ 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "iostat", 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "nfs", 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "restful" 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "servicemap": { 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modified": "2024-03-21T21:13:16.822749+0000", 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-03-21T21:13:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-21T21:13:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-03-21T21:13:33.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:33 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3547241059' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T21:13:33.183 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:33.061+0000 7fdfc5ae3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:13:33.183 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:33.156+0000 7fdfc5ae3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:13:33.659 INFO:teuthology.orchestra.run.smithi028.stderr:mgr not available, waiting (1/15)... 2024-03-21T21:13:34.080 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:33.770+0000 7fdfc5ae3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:13:34.080 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:33.860+0000 7fdfc5ae3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:13:34.080 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:33.943+0000 7fdfc5ae3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:13:34.355 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:34.092+0000 7fdfc5ae3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:13:34.355 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:34.173+0000 7fdfc5ae3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:13:34.356 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:34.284+0000 7fdfc5ae3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:13:34.356 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:34.367+0000 7fdfc5ae3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:13:34.682 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:34.530+0000 7fdfc5ae3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:13:34.682 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:34.616+0000 7fdfc5ae3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:13:35.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:35.687+0000 7fdfc5ae3000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:13:35.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:35.823+0000 7fdfc5ae3000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:13:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsid": "c10cc158-e7c7-11ee-95cd-87774f69a715", 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "health": { 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "checks": {}, 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mutes": [] 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum": [ 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 0 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "a" 2024-03-21T21:13:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "monmap": { 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osdmap": { 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-21T21:13:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgmap": { 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-21T21:13:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsmap": { 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": false, 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modules": [ 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "iostat", 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "nfs", 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "restful" 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-03-21T21:13:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "servicemap": { 2024-03-21T21:13:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modified": "2024-03-21T21:13:16.822749+0000", 2024-03-21T21:13:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-03-21T21:13:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-21T21:13:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-03-21T21:13:36.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:36 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1366975913' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T21:13:36.433 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:36 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:36.019+0000 7fdfc5ae3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:13:36.433 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:36 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:36.111+0000 7fdfc5ae3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:13:36.433 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:36 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:36.253+0000 7fdfc5ae3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:13:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:mgr not available, waiting (2/15)... 2024-03-21T21:13:36.873 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:36 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:36.575+0000 7fdfc5ae3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:13:36.873 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:36 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:36.799+0000 7fdfc5ae3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:13:36.873 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:36 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:36.885+0000 7fdfc5ae3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:13:37.182 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:37 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:37.059+0000 7fdfc5ae3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:13:38.279 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:37 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:37.991+0000 7fdfc5ae3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:13:38.279 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:38.291+0000 7fdfc5ae3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:13:38.683 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:38.387+0000 7fdfc5ae3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:13:38.683 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:38.539+0000 7fdfc5ae3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:13:39.186 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsid": "c10cc158-e7c7-11ee-95cd-87774f69a715", 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "health": { 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "checks": {}, 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mutes": [] 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum": [ 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 0 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-21T21:13:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "a" 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "monmap": { 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osdmap": { 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-21T21:13:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-21T21:13:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-21T21:13:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-21T21:13:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-21T21:13:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgmap": { 2024-03-21T21:13:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-21T21:13:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-21T21:13:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsmap": { 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-21T21:13:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": false, 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modules": [ 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "iostat", 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "nfs", 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "restful" 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "servicemap": { 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modified": "2024-03-21T21:13:16.822749+0000", 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-03-21T21:13:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-21T21:13:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: Activating manager daemon y 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: mgrmap e2: y(active, starting, since 0.00275933s) 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: Manager daemon y is now available 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' 2024-03-21T21:13:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:39 smithi028 ceph-mon[24806]: from='mgr.14100 172.21.15.28:0/445288301' entity='mgr.y' 2024-03-21T21:13:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:mgr not available, waiting (3/15)... 2024-03-21T21:13:40.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:40 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1689487646' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T21:13:40.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:40 smithi028 ceph-mon[24806]: mgrmap e3: y(active, since 1.00552s) 2024-03-21T21:13:42.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:42 smithi028 ceph-mon[24806]: mgrmap e4: y(active, since 2s) 2024-03-21T21:13:42.391 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsid": "c10cc158-e7c7-11ee-95cd-87774f69a715", 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "health": { 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "checks": {}, 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mutes": [] 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum": [ 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 0 2024-03-21T21:13:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "a" 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_age": 14, 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "monmap": { 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osdmap": { 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-21T21:13:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-21T21:13:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-21T21:13:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-21T21:13:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-21T21:13:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-21T21:13:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgmap": { 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsmap": { 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": true, 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modules": [ 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "iostat", 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "nfs", 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "restful" 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "servicemap": { 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modified": "2024-03-21T21:13:16.822749+0000", 2024-03-21T21:13:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-03-21T21:13:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-03-21T21:13:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-21T21:13:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-03-21T21:13:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:mgr is available 2024-03-21T21:13:43.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:43 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1075838690' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [global] 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: fsid = c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [mgr] 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [osd] 2024-03-21T21:13:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-21T21:13:43.664 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-21T21:13:44.135 INFO:teuthology.orchestra.run.smithi028.stderr:Enabling cephadm module... 2024-03-21T21:13:44.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:44 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/459490300' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T21:13:44.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:44 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/459490300' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T21:13:45.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:45 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1003275286' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-21T21:13:45.422 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ignoring --setuser ceph since I am not root 2024-03-21T21:13:45.423 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ignoring --setgroup ceph since I am not root 2024-03-21T21:13:45.680 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:45.642+0000 7ff36476d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:13:45.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:45.737+0000 7ff36476d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:13:46.429 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:46.351+0000 7ff36476d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:13:46.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:46 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1003275286' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-21T21:13:46.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:46 smithi028 ceph-mon[24806]: mgrmap e5: y(active, since 6s) 2024-03-21T21:13:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-03-21T21:13:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-21T21:13:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": true, 2024-03-21T21:13:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-21T21:13:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-21T21:13:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-03-21T21:13:46.683 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:46.442+0000 7ff36476d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:13:46.683 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:46.531+0000 7ff36476d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:13:46.683 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:46.682+0000 7ff36476d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:13:47.028 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:46.764+0000 7ff36476d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:13:47.028 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:46.876+0000 7ff36476d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:13:47.028 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:46.961+0000 7ff36476d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:13:47.030 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for the mgr to restart... 2024-03-21T21:13:47.030 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mgr epoch 5... 2024-03-21T21:13:47.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:47 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/94961695' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T21:13:47.331 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:47.128+0000 7ff36476d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:13:47.331 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:47.217+0000 7ff36476d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:13:48.576 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:48.264+0000 7ff36476d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:13:48.576 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:48.396+0000 7ff36476d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:13:48.576 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:48.589+0000 7ff36476d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:13:48.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:48.677+0000 7ff36476d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:13:48.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:48.815+0000 7ff36476d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:13:49.432 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:49 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:49.134+0000 7ff36476d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:13:49.432 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:49 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:49.359+0000 7ff36476d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:13:49.432 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:49 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:49.445+0000 7ff36476d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:13:49.683 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:49 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:49.618+0000 7ff36476d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:13:50.841 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:50.550+0000 7ff36476d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:13:51.183 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:50.854+0000 7ff36476d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:13:51.183 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:50.950+0000 7ff36476d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:13:51.183 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:51 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:13:51.103+0000 7ff36476d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:13:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-03-21T21:13:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-21T21:13:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "initialized": true 2024-03-21T21:13:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-03-21T21:13:52.433 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:13:52] ENGINE Bus STARTING 2024-03-21T21:13:52.434 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:13:52] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:13:52.434 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:13:52] ENGINE Bus STARTED 2024-03-21T21:13:52.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: Active manager daemon y restarted 2024-03-21T21:13:52.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: Activating manager daemon y 2024-03-21T21:13:52.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: osdmap e2: 0 total, 0 up, 0 in 2024-03-21T21:13:52.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: mgrmap e6: y(active, starting, since 0.0525157s) 2024-03-21T21:13:52.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:13:52.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:13:52.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:13:52.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:13:52.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:13:52.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: Manager daemon y is now available 2024-03-21T21:13:52.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:13:52.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:13:52.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:13:52.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:13:52.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:13:52.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:13:52.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:52 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:13:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:mgr epoch 5 is available 2024-03-21T21:13:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:Setting orchestrator backend to cephadm... 2024-03-21T21:13:53.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:53 smithi028 ceph-mon[24806]: [21/Mar/2024:21:13:52] ENGINE Bus STARTING 2024-03-21T21:13:53.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:53 smithi028 ceph-mon[24806]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T21:13:53.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:53 smithi028 ceph-mon[24806]: mgrmap e7: y(active, since 1.05463s) 2024-03-21T21:13:53.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:53 smithi028 ceph-mon[24806]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T21:13:53.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:53 smithi028 ceph-mon[24806]: [21/Mar/2024:21:13:52] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:13:53.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:53 smithi028 ceph-mon[24806]: [21/Mar/2024:21:13:52] ENGINE Bus STARTED 2024-03-21T21:13:53.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:53 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:13:53.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:53 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:13:54.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:54 smithi028 ceph-mon[24806]: mgrmap e8: y(active, since 2s) 2024-03-21T21:13:54.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:54 smithi028 ceph-mon[24806]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:13:54.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:54 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:13:54.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:54 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:13:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: value unchanged 2024-03-21T21:13:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:Generating ssh key... 2024-03-21T21:13:55.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-mon[24806]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:13:56.384 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: Generating public/private rsa key pair. 2024-03-21T21:13:56.384 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: Your identification has been saved in /tmp/tmp1ia05pas/key. 2024-03-21T21:13:56.384 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: Your public key has been saved in /tmp/tmp1ia05pas/key.pub. 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: The key fingerprint is: 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: SHA256:GSvSWqM2DAV+ld8+n1o+8GcQ/U2vFTg4dsyTiafIZ1k ceph-c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: The key's randomart image is: 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: +---[RSA 3072]----+ 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: | . .. | 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: | . . .. | 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: | . o ... =.+ | 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: | o . .+.=.E...| 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: | . . = S.o B.oo+| 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: | o = o o+=. =| 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: | * o=.o o | 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: | . . o= + | 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: | ...+ | 2024-03-21T21:13:56.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:13:55 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: +----[SHA256]-----+ 2024-03-21T21:13:56.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:56 smithi028 ceph-mon[24806]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:13:56.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:56 smithi028 ceph-mon[24806]: Generating ssh key... 2024-03-21T21:13:56.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:56 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:13:56.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:56 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:13:56.997 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNhQ1q8tnTDfB/QmEBQDR1x/iR770tseLjL5tUccR+VnXFW2iZSDxuAoIG/VQRqmBdiEJkTlJ4932anCMqjpXF/gGMhBu1AmqmQAdASdUOXACwFghvU/gfF+dMmhW1BYz8SnnTmWyDdvAwIeN1xNXO9PEPzNxpOFuPztmhxwSfJuAON9oo25j4obMtCQOZtEE0Z58vdoWrI3KVXdRnXFSvogYyuKuTBDSvnz5Wi1XnwLV2+7gulhpMYueHGqpFnz5xWkLtewtDFDquMo8P1FKNaR+8nU3rB/+ZMTkFFJoAaFx0vksrgiD9guz4b0EJpbGNQaH1L9S3A6GqJpVxlKzh2VeMIaSdvhD/fwm3hxolc7YT3jpR1TNeKvTRqPLqua0FqXgUpIp1Iv+/JccakgZ4EFvSkvXPb8g9nGNtZ5hynMUdUK+Ij6IbHZEXS+f5u/oQJcqXzWyCegFVX3fG5NyAE/EXyWqbDCY/NDSHt2JGztwYIt+naYKlDdrI7arl60= ceph-c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:13:57.576 INFO:teuthology.orchestra.run.smithi028.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-21T21:13:57.576 INFO:teuthology.orchestra.run.smithi028.stderr:Adding key to root@localhost authorized_keys... 2024-03-21T21:13:57.577 INFO:teuthology.orchestra.run.smithi028.stderr:Adding host smithi028... 2024-03-21T21:13:58.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:57 smithi028 ceph-mon[24806]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:13:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Added host 'smithi028' with addr '172.21.15.28' 2024-03-21T21:13:59.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:59 smithi028 ceph-mon[24806]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:13:59.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:59 smithi028 ceph-mon[24806]: Deploying cephadm binary to smithi028 2024-03-21T21:13:59.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:59 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:13:59.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:13:59 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:13:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:Deploying unmanaged mon service... 2024-03-21T21:14:00.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:00 smithi028 ceph-mon[24806]: Added host smithi028 2024-03-21T21:14:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-21T21:14:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:Deploying unmanaged mgr service... 2024-03-21T21:14:01.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:01 smithi028 ceph-mon[24806]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:14:01.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:01 smithi028 ceph-mon[24806]: Saving service mon spec with placement count:5 2024-03-21T21:14:01.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:01 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:14:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-21T21:14:02.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:02 smithi028 ceph-mon[24806]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:14:02.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:02 smithi028 ceph-mon[24806]: Saving service mgr spec with placement count:2 2024-03-21T21:14:02.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:02 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:14:04.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:03 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:14:04.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:03 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:14:05.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:05 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2177741339' entity='client.admin' 2024-03-21T21:14:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:Enabling the dashboard module... 2024-03-21T21:14:06.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:06 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:14:06.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:06 smithi028 ceph-mon[24806]: from='mgr.14120 172.21.15.28:0/3109905034' entity='mgr.y' 2024-03-21T21:14:06.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:06 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3744447337' entity='client.admin' 2024-03-21T21:14:07.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:07 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/228854880' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-21T21:14:07.434 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:07 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ignoring --setuser ceph since I am not root 2024-03-21T21:14:07.434 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:07 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ignoring --setgroup ceph since I am not root 2024-03-21T21:14:07.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:07 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:07.685+0000 7f5e45f13000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:14:07.934 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:07 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:07.776+0000 7f5e45f13000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:14:08.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:08 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/228854880' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-21T21:14:08.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:08 smithi028 ceph-mon[24806]: mgrmap e9: y(active, since 16s) 2024-03-21T21:14:08.656 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:08.383+0000 7f5e45f13000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:14:08.656 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:08.482+0000 7f5e45f13000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:14:08.657 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:08.569+0000 7f5e45f13000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:14:08.910 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:08.721+0000 7f5e45f13000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:14:08.910 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:08.805+0000 7f5e45f13000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:14:08.910 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:08.921+0000 7f5e45f13000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:14:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-03-21T21:14:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-21T21:14:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": true, 2024-03-21T21:14:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-21T21:14:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-21T21:14:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-03-21T21:14:09.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:09 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2298478531' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T21:14:09.164 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:09 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:09.007+0000 7f5e45f13000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:14:09.164 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:09 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:09.175+0000 7f5e45f13000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:14:09.434 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:09 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:09.262+0000 7f5e45f13000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:14:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for the mgr to restart... 2024-03-21T21:14:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mgr epoch 9... 2024-03-21T21:14:10.628 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:10.313+0000 7f5e45f13000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:14:10.628 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:10.445+0000 7f5e45f13000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:14:10.628 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:10.639+0000 7f5e45f13000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:14:10.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:10.728+0000 7f5e45f13000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:14:10.934 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:10.866+0000 7f5e45f13000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:14:11.433 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:11.187+0000 7f5e45f13000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:14:11.434 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:11.412+0000 7f5e45f13000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:14:11.934 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:11.499+0000 7f5e45f13000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:14:11.934 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:11.673+0000 7f5e45f13000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:14:12.895 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:12.605+0000 7f5e45f13000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:14:12.895 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:12.906+0000 7f5e45f13000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:14:13.183 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:13.002+0000 7f5e45f13000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:14:13.184 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:14:13.156+0000 7f5e45f13000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: Active manager daemon y restarted 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: Activating manager daemon y 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: osdmap e3: 0 total, 0 up, 0 in 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: mgrmap e10: y(active, starting, since 0.967932s) 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:14:14.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-mon[24806]: Manager daemon y is now available 2024-03-21T21:14:14.875 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:14:14] ENGINE Bus STARTING 2024-03-21T21:14:14.875 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:14:14] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:14:14.875 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:14:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:14:14] ENGINE Bus STARTED 2024-03-21T21:14:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-03-21T21:14:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-21T21:14:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "initialized": true 2024-03-21T21:14:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-03-21T21:14:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:14:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:14:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:14:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:14:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:15 smithi028 ceph-mon[24806]: mgrmap e11: y(active, since 1.96984s) 2024-03-21T21:14:16.030 INFO:teuthology.orchestra.run.smithi028.stderr:mgr epoch 9 is available 2024-03-21T21:14:16.031 INFO:teuthology.orchestra.run.smithi028.stderr:Generating a dashboard self-signed certificate... 2024-03-21T21:14:16.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:16 smithi028 ceph-mon[24806]: [21/Mar/2024:21:14:14] ENGINE Bus STARTING 2024-03-21T21:14:16.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:16 smithi028 ceph-mon[24806]: [21/Mar/2024:21:14:14] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:14:16.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:16 smithi028 ceph-mon[24806]: [21/Mar/2024:21:14:14] ENGINE Bus STARTED 2024-03-21T21:14:16.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:16 smithi028 ceph-mon[24806]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T21:14:16.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:16 smithi028 ceph-mon[24806]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T21:14:16.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:16 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:16.706 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-21T21:14:17.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:17 smithi028 ceph-mon[24806]: mgrmap e12: y(active, since 3s) 2024-03-21T21:14:17.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:17.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:17.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:Creating initial admin user... 2024-03-21T21:14:18.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:18 smithi028 ceph-mon[24806]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:14:18.417 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$.laVYwCzdIYEeKeTXH4S1OBGIu2Jvr.Mq41q1oSTVQideb2hZKnRG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711055658, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-21T21:14:19.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:19 smithi028 ceph-mon[24806]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:14:19.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:19 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:19.566 INFO:teuthology.orchestra.run.smithi028.stderr:Fetching dashboard port number... 2024-03-21T21:14:20.238 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 8443 2024-03-21T21:14:20.901 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: disabled 2024-03-21T21:14:20.908 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-03-21T21:14:20.909 INFO:teuthology.orchestra.run.smithi028.stderr:firewalld.service is not enabled 2024-03-21T21:14:20.909 INFO:teuthology.orchestra.run.smithi028.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-21T21:14:20.911 INFO:teuthology.orchestra.run.smithi028.stderr:Ceph Dashboard is now available at: 2024-03-21T21:14:20.912 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:14:20.912 INFO:teuthology.orchestra.run.smithi028.stderr: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-03-21T21:14:20.912 INFO:teuthology.orchestra.run.smithi028.stderr: User: admin 2024-03-21T21:14:20.912 INFO:teuthology.orchestra.run.smithi028.stderr: Password: 145zeoefpi 2024-03-21T21:14:20.912 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:14:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:Enabling autotune for osd_memory_target 2024-03-21T21:14:21.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:21 smithi028 ceph-mon[24806]: mgrmap e13: y(active, since 7s) 2024-03-21T21:14:21.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:21 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1536847281' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-21T21:14:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-21T21:14:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:You can access the Ceph CLI with: 2024-03-21T21:14:23.570 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:14:23.570 INFO:teuthology.orchestra.run.smithi028.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-21T21:14:23.570 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:14:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-21T21:14:23.570 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:14:23.570 INFO:teuthology.orchestra.run.smithi028.stderr: ceph telemetry on 2024-03-21T21:14:23.571 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:14:23.571 INFO:teuthology.orchestra.run.smithi028.stderr:For more information see: 2024-03-21T21:14:23.571 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:14:23.571 INFO:teuthology.orchestra.run.smithi028.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-21T21:14:23.571 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:14:23.571 INFO:teuthology.orchestra.run.smithi028.stderr:Bootstrap complete. 2024-03-21T21:14:23.609 INFO:tasks.cephadm:Fetching config... 2024-03-21T21:14:23.609 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:14:23.609 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-21T21:14:23.626 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-21T21:14:23.626 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:14:23.626 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-21T21:14:23.680 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-21T21:14:23.681 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:14:23.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-21T21:14:23.747 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-21T21:14:23.747 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:14:23.747 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-21T21:14:23.802 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-21T21:14:23.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNhQ1q8tnTDfB/QmEBQDR1x/iR770tseLjL5tUccR+VnXFW2iZSDxuAoIG/VQRqmBdiEJkTlJ4932anCMqjpXF/gGMhBu1AmqmQAdASdUOXACwFghvU/gfF+dMmhW1BYz8SnnTmWyDdvAwIeN1xNXO9PEPzNxpOFuPztmhxwSfJuAON9oo25j4obMtCQOZtEE0Z58vdoWrI3KVXdRnXFSvogYyuKuTBDSvnz5Wi1XnwLV2+7gulhpMYueHGqpFnz5xWkLtewtDFDquMo8P1FKNaR+8nU3rB/+ZMTkFFJoAaFx0vksrgiD9guz4b0EJpbGNQaH1L9S3A6GqJpVxlKzh2VeMIaSdvhD/fwm3hxolc7YT3jpR1TNeKvTRqPLqua0FqXgUpIp1Iv+/JccakgZ4EFvSkvXPb8g9nGNtZ5hynMUdUK+Ij6IbHZEXS+f5u/oQJcqXzWyCegFVX3fG5NyAE/EXyWqbDCY/NDSHt2JGztwYIt+naYKlDdrI7arl60= ceph-c10cc158-e7c7-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T21:14:23.880 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNhQ1q8tnTDfB/QmEBQDR1x/iR770tseLjL5tUccR+VnXFW2iZSDxuAoIG/VQRqmBdiEJkTlJ4932anCMqjpXF/gGMhBu1AmqmQAdASdUOXACwFghvU/gfF+dMmhW1BYz8SnnTmWyDdvAwIeN1xNXO9PEPzNxpOFuPztmhxwSfJuAON9oo25j4obMtCQOZtEE0Z58vdoWrI3KVXdRnXFSvogYyuKuTBDSvnz5Wi1XnwLV2+7gulhpMYueHGqpFnz5xWkLtewtDFDquMo8P1FKNaR+8nU3rB/+ZMTkFFJoAaFx0vksrgiD9guz4b0EJpbGNQaH1L9S3A6GqJpVxlKzh2VeMIaSdvhD/fwm3hxolc7YT3jpR1TNeKvTRqPLqua0FqXgUpIp1Iv+/JccakgZ4EFvSkvXPb8g9nGNtZ5hynMUdUK+Ij6IbHZEXS+f5u/oQJcqXzWyCegFVX3fG5NyAE/EXyWqbDCY/NDSHt2JGztwYIt+naYKlDdrI7arl60= ceph-c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:14:23.894 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNhQ1q8tnTDfB/QmEBQDR1x/iR770tseLjL5tUccR+VnXFW2iZSDxuAoIG/VQRqmBdiEJkTlJ4932anCMqjpXF/gGMhBu1AmqmQAdASdUOXACwFghvU/gfF+dMmhW1BYz8SnnTmWyDdvAwIeN1xNXO9PEPzNxpOFuPztmhxwSfJuAON9oo25j4obMtCQOZtEE0Z58vdoWrI3KVXdRnXFSvogYyuKuTBDSvnz5Wi1XnwLV2+7gulhpMYueHGqpFnz5xWkLtewtDFDquMo8P1FKNaR+8nU3rB/+ZMTkFFJoAaFx0vksrgiD9guz4b0EJpbGNQaH1L9S3A6GqJpVxlKzh2VeMIaSdvhD/fwm3hxolc7YT3jpR1TNeKvTRqPLqua0FqXgUpIp1Iv+/JccakgZ4EFvSkvXPb8g9nGNtZ5hynMUdUK+Ij6IbHZEXS+f5u/oQJcqXzWyCegFVX3fG5NyAE/EXyWqbDCY/NDSHt2JGztwYIt+naYKlDdrI7arl60= ceph-c10cc158-e7c7-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T21:14:23.949 INFO:teuthology.orchestra.run.smithi066.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNhQ1q8tnTDfB/QmEBQDR1x/iR770tseLjL5tUccR+VnXFW2iZSDxuAoIG/VQRqmBdiEJkTlJ4932anCMqjpXF/gGMhBu1AmqmQAdASdUOXACwFghvU/gfF+dMmhW1BYz8SnnTmWyDdvAwIeN1xNXO9PEPzNxpOFuPztmhxwSfJuAON9oo25j4obMtCQOZtEE0Z58vdoWrI3KVXdRnXFSvogYyuKuTBDSvnz5Wi1XnwLV2+7gulhpMYueHGqpFnz5xWkLtewtDFDquMo8P1FKNaR+8nU3rB/+ZMTkFFJoAaFx0vksrgiD9guz4b0EJpbGNQaH1L9S3A6GqJpVxlKzh2VeMIaSdvhD/fwm3hxolc7YT3jpR1TNeKvTRqPLqua0FqXgUpIp1Iv+/JccakgZ4EFvSkvXPb8g9nGNtZ5hynMUdUK+Ij6IbHZEXS+f5u/oQJcqXzWyCegFVX3fG5NyAE/EXyWqbDCY/NDSHt2JGztwYIt+naYKlDdrI7arl60= ceph-c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:14:23.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-21T21:14:24.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:24 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2754599431' entity='client.admin' 2024-03-21T21:14:28.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:28 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/4266956223' entity='client.admin' 2024-03-21T21:14:28.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:28 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:28.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:28 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:14:28.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:28 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:28.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:28 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:14:28.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:28 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:28.255 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-21T21:14:28.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-21T21:14:30.873 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi066 2024-03-21T21:14:30.873 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-21T21:14:30.873 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/etc/ceph/ceph.conf 2024-03-21T21:14:30.892 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-21T21:14:30.892 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:14:30.950 INFO:tasks.cephadm:Adding host smithi066 to orchestrator... 2024-03-21T21:14:30.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch host add smithi066 2024-03-21T21:14:31.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:31 smithi028 ceph-mon[24806]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:14:31.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:31.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:14:31.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:14:31.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:14:31.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:31 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:14:31.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:31.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:32.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:32 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:14:33.939 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi066' with addr '172.21.15.66' 2024-03-21T21:14:34.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:34 smithi028 ceph-mon[24806]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:14:34.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:34 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:34.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:34 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:14:34.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:34 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:14:34.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:34 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:14:34.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-21T21:14:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:35 smithi028 ceph-mon[24806]: Deploying cephadm binary to smithi066 2024-03-21T21:14:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:35 smithi028 ceph-mon[24806]: Added host smithi066 2024-03-21T21:14:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:35 smithi028 ceph-mon[24806]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:35 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:35 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:36.309 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:14:36.309 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.66", "hostname": "smithi066", "labels": [], "status": ""}] 2024-03-21T21:14:36.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:36 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:36.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:36 smithi028 ceph-mon[24806]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:36.913 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-21T21:14:36.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-21T21:14:37.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:37 smithi028 ceph-mon[24806]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T21:14:39.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:39 smithi028 ceph-mon[24806]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:39.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi066", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:14:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:39 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3194213849' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T21:14:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-03-21T21:14:40.123 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-03-21T21:14:40.123 INFO:tasks.cephadm:Adding mon.c on smithi028 2024-03-21T21:14:40.123 INFO:tasks.cephadm:Adding mon.b on smithi066 2024-03-21T21:14:40.123 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi028:[v2:172.21.15.28:3301,v1:172.21.15.28:6790]=c;smithi066:172.21.15.66=b' 2024-03-21T21:14:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:40 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3194213849' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T21:14:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:40 smithi028 ceph-mon[24806]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T21:14:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:40 smithi028 ceph-mon[24806]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:42.008 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled mon update... 2024-03-21T21:14:42.698 DEBUG:teuthology.orchestra.run.smithi028:mon.c> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.c.service 2024-03-21T21:14:42.700 DEBUG:teuthology.orchestra.run.smithi066:mon.b> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.b.service 2024-03-21T21:14:42.702 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T21:14:42.703 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T21:14:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi028:[v2:172.21.15.28:3301,v1:172.21.15.28:6790]=c;smithi066:172.21.15.66=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi028:[v2:172.21.15.28:3301,v1:172.21.15.28:6790]=c;smithi066:172.21.15.66=b;count:3 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:14:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:43 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:14:44.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:44 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:14:44.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:44 smithi028 ceph-mon[24806]: Deploying daemon mon.c on smithi028 2024-03-21T21:14:44.605 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:14:44.606 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c10cc158-e7c7-11ee-95cd-87774f69a715","modified":"2024-03-21T21:13:11.624978Z","created":"2024-03-21T21:13:11.624978Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-21T21:14:44.608 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-03-21T21:14:45.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:45 smithi028 ceph-mon[24806]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:45.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:45 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/91375023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T21:14:46.252 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T21:14:46.253 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T21:14:51.185 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:14:51.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:14:51.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: mon.a calling monitor election 2024-03-21T21:14:51.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:14:51.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:51.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:14:51.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:14:51.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: mon.c calling monitor election 2024-03-21T21:14:51.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:51.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: monmap e2: 2 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: fsmap 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: mgrmap e13: y(active, since 37s) 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: overall HEALTH_OK 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:14:51.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:14:51.267 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:14:51.267 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":2,"fsid":"c10cc158-e7c7-11ee-95cd-87774f69a715","modified":"2024-03-21T21:14:45.823096Z","created":"2024-03-21T21:13:11.624978Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3301","nonce":0},{"type":"v1","addr":"172.21.15.28:6790","nonce":0}]},"addr":"172.21.15.28:6790/0","public_addr":"172.21.15.28:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T21:14:51.268 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 2 2024-03-21T21:14:52.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:51 smithi028 ceph-mon[24806]: Deploying daemon mon.b on smithi066 2024-03-21T21:14:52.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:51 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/464135079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T21:14:52.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:51 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:14:52.185 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:14:51 smithi028 ceph-mon[29568]: Deploying daemon mon.b on smithi066 2024-03-21T21:14:52.185 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:14:51 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/464135079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T21:14:52.185 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:14:51 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:14:52.964 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T21:14:52.964 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T21:14:53.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:52 smithi028 ceph-mon[24806]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:53.185 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:14:52 smithi028 ceph-mon[29568]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:55.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:14:55 smithi028 ceph-mon[24806]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:55.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:14:55 smithi028 ceph-mon[29568]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:14:55.826 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:14:55.826 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":2,"fsid":"c10cc158-e7c7-11ee-95cd-87774f69a715","modified":"2024-03-21T21:14:45.823096Z","created":"2024-03-21T21:13:11.624978Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3301","nonce":0},{"type":"v1","addr":"172.21.15.28:6790","nonce":0}]},"addr":"172.21.15.28:6790/0","public_addr":"172.21.15.28:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T21:14:55.827 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 2 2024-03-21T21:14:57.439 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T21:14:57.440 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T21:15:01.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:15:01.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: mon.c calling monitor election 2024-03-21T21:15:01.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: mon.a calling monitor election 2024-03-21T21:15:01.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:15:01.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:01.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:01.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: fsmap 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: mgrmap e13: y(active, since 47s) 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: mon.b (rank 2) addr [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] is down (out of quorum) 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:01.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: mon.c calling monitor election 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: mon.a calling monitor election 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: fsmap 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: mgrmap e13: y(active, since 47s) 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: mon.b (rank 2) addr [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] is down (out of quorum) 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:02.286 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-21T21:15:02.286 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":3,"fsid":"c10cc158-e7c7-11ee-95cd-87774f69a715","modified":"2024-03-21T21:14:55.844813Z","created":"2024-03-21T21:13:11.624978Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3301","nonce":0},{"type":"v1","addr":"172.21.15.28:6790","nonce":0}]},"addr":"172.21.15.28:6790/0","public_addr":"172.21.15.28:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:3300","nonce":0},{"type":"v1","addr":"172.21.15.66:6789","nonce":0}]},"addr":"172.21.15.66:6789/0","public_addr":"172.21.15.66:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T21:15:02.287 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 3 2024-03-21T21:15:02.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:15:02.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:02.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:15:02.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: Reconfiguring daemon mon.a on smithi028 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/2377284322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:15:02.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: Reconfiguring daemon mon.a on smithi028 2024-03-21T21:15:02.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:02 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/2377284322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T21:15:02.875 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-21T21:15:02.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-21T21:15:04.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: mon.b calling monitor election 2024-03-21T21:15:04.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: mon.b calling monitor election 2024-03-21T21:15:04.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: mon.a calling monitor election 2024-03-21T21:15:04.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: mon.c calling monitor election 2024-03-21T21:15:04.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:15:04.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} 2024-03-21T21:15:04.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: fsmap 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: mgrmap e13: y(active, since 49s) 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: Cluster is now healthy 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[24806]: overall HEALTH_OK 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: mon.b calling monitor election 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: mon.b calling monitor election 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: mon.a calling monitor election 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: mon.c calling monitor election 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:15:04.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} 2024-03-21T21:15:04.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: fsmap 2024-03-21T21:15:04.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T21:15:04.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: mgrmap e13: y(active, since 49s) 2024-03-21T21:15:04.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-21T21:15:04.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: Cluster is now healthy 2024-03-21T21:15:04.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:03 smithi028 ceph-mon[29568]: overall HEALTH_OK 2024-03-21T21:15:04.238 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:03 smithi066 ceph-mon[25664]: overall HEALTH_OK 2024-03-21T21:15:04.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:04.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:04.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[24806]: Reconfiguring mon.c (monmap changed)... 2024-03-21T21:15:04.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:15:04.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:15:04.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:04.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[24806]: Reconfiguring daemon mon.c on smithi028 2024-03-21T21:15:04.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[24806]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:04.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[29568]: Reconfiguring mon.c (monmap changed)... 2024-03-21T21:15:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:15:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:15:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[29568]: Reconfiguring daemon mon.c on smithi028 2024-03-21T21:15:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[29568]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:04 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:05.238 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:04 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:15:05.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:04 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:05.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:04 smithi066 ceph-mon[25664]: Reconfiguring mon.c (monmap changed)... 2024-03-21T21:15:05.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:04 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:15:05.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:04 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:15:05.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:04 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:05.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:04 smithi066 ceph-mon[25664]: Reconfiguring daemon mon.c on smithi028 2024-03-21T21:15:05.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:04 smithi066 ceph-mon[25664]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:05.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:04 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:05.864 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:15:05.864 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-03-21T21:15:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:15:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] [v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0] 2024-03-21T21:15:06.696 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-21T21:15:06.696 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:15:06.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T21:15:06.726 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:15:06.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:15:06.793 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-21T21:15:06.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T21:15:06.825 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-21T21:15:06.826 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:15:06.897 INFO:tasks.cephadm:Adding mgr.y on smithi028 2024-03-21T21:15:06.897 INFO:tasks.cephadm:Adding mgr.x on smithi066 2024-03-21T21:15:06.897 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi028=y;smithi066=x' 2024-03-21T21:15:06.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1264250650' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:06.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[24806]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:06.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:06.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[24806]: Reconfiguring mon.b (monmap changed)... 2024-03-21T21:15:06.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:15:06.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:15:06.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:06.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[24806]: Reconfiguring daemon mon.b on smithi066 2024-03-21T21:15:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1264250650' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[29568]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[29568]: Reconfiguring mon.b (monmap changed)... 2024-03-21T21:15:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:15:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:15:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:06 smithi028 ceph-mon[29568]: Reconfiguring daemon mon.b on smithi066 2024-03-21T21:15:07.238 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:06 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1264250650' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:07.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:06 smithi066 ceph-mon[25664]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:07.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:06 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:07.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:06 smithi066 ceph-mon[25664]: Reconfiguring mon.b (monmap changed)... 2024-03-21T21:15:07.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:06 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:15:07.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:06 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:15:07.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:06 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:07.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:06 smithi066 ceph-mon[25664]: Reconfiguring daemon mon.b on smithi066 2024-03-21T21:15:08.739 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:08 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:08.739 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:08 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:08.739 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:08 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:08.739 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:08 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:08.739 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:08 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:08.739 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:08 smithi066 ceph-mon[25664]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:08.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:08.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:08.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:08.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:08.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:08.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[29568]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:08.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:08.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:08.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:08.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:08.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:08.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:08 smithi028 ceph-mon[24806]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:09.063 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled mgr update... 2024-03-21T21:15:09.638 DEBUG:teuthology.orchestra.run.smithi066:mgr.x> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service 2024-03-21T21:15:09.640 INFO:tasks.cephadm:Deploying OSDs... 2024-03-21T21:15:09.640 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:15:09.640 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T21:15:09.659 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T21:15:09.659 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-03-21T21:15:09.717 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T21:15:09.717 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T21:15:09.717 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-21T21:15:09.717 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T21:15:09.717 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T21:15:09.717 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-21 21:14:45.283536584 +0000 2024-03-21T21:15:09.717 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-21 21:12:18.725277147 +0000 2024-03-21T21:15:09.717 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-21 21:12:18.725277147 +0000 2024-03-21T21:15:09.717 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-21 21:12:18.725277147 +0000 2024-03-21T21:15:09.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T21:15:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-21T21:15:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-21T21:15:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000120848 s, 4.2 MB/s 2024-03-21T21:15:09.789 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T21:15:09.847 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-03-21T21:15:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T21:15:09.904 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T21:15:09.904 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-21T21:15:09.904 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T21:15:09.904 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T21:15:09.904 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-21 21:14:45.284536557 +0000 2024-03-21T21:15:09.905 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-21 21:12:19.053268297 +0000 2024-03-21T21:15:09.905 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-21 21:12:19.053268297 +0000 2024-03-21T21:15:09.905 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-21 21:12:19.053268297 +0000 2024-03-21T21:15:09.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T21:15:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-21T21:15:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-21T21:15:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000100554 s, 5.1 MB/s 2024-03-21T21:15:09.974 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T21:15:10.031 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-03-21T21:15:10.088 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T21:15:10.088 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T21:15:10.088 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-21T21:15:10.088 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T21:15:10.088 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T21:15:10.089 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-21 21:14:45.284536557 +0000 2024-03-21T21:15:10.089 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-21 21:12:19.399258961 +0000 2024-03-21T21:15:10.089 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-21 21:12:19.399258961 +0000 2024-03-21T21:15:10.089 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-21 21:12:19.399258961 +0000 2024-03-21T21:15:10.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi066=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: Saving service mgr spec with placement smithi028=y;smithi066=x;count:2 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:10.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:10 smithi066 ceph-mon[25664]: Deploying daemon mgr.x on smithi066 2024-03-21T21:15:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-21T21:15:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-21T21:15:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000105351 s, 4.9 MB/s 2024-03-21T21:15:10.156 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T21:15:10.213 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-03-21T21:15:10.268 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T21:15:10.268 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T21:15:10.268 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-21T21:15:10.268 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T21:15:10.268 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T21:15:10.268 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-21 21:14:45.285536530 +0000 2024-03-21T21:15:10.269 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-21 21:12:19.739249788 +0000 2024-03-21T21:15:10.269 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-21 21:12:19.739249788 +0000 2024-03-21T21:15:10.269 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-21 21:12:19.739249788 +0000 2024-03-21T21:15:10.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T21:15:10.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi066=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: Saving service mgr spec with placement smithi028=y;smithi066=x;count:2 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[24806]: Deploying daemon mgr.x on smithi066 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi066=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: Saving service mgr spec with placement smithi028=y;smithi066=x;count:2 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:10 smithi028 ceph-mon[29568]: Deploying daemon mgr.x on smithi066 2024-03-21T21:15:10.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-21T21:15:10.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-21T21:15:10.335 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 9.9538e-05 s, 5.1 MB/s 2024-03-21T21:15:10.336 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T21:15:10.393 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-21T21:15:10.393 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T21:15:10.410 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T21:15:10.411 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 2024-03-21T21:15:10.467 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T21:15:10.468 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T21:15:10.468 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-21T21:15:10.468 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T21:15:10.468 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T21:15:10.468 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-21 21:14:54.670589781 +0000 2024-03-21T21:15:10.468 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-21 21:12:12.043255543 +0000 2024-03-21T21:15:10.468 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-21 21:12:12.043255543 +0000 2024-03-21T21:15:10.468 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-21 21:12:12.043255543 +0000 2024-03-21T21:15:10.468 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T21:15:10.540 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-21T21:15:10.540 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-21T21:15:10.540 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.00019433 s, 2.6 MB/s 2024-03-21T21:15:10.541 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T21:15:10.599 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 2024-03-21T21:15:10.655 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T21:15:10.655 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T21:15:10.655 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-21T21:15:10.655 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T21:15:10.655 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T21:15:10.656 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-21 21:14:54.670589781 +0000 2024-03-21T21:15:10.656 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-21 21:12:12.394252464 +0000 2024-03-21T21:15:10.656 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-21 21:12:12.394252464 +0000 2024-03-21T21:15:10.656 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-21 21:12:12.394252464 +0000 2024-03-21T21:15:10.656 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T21:15:10.857 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-21T21:15:10.857 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-21T21:15:10.857 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000177787 s, 2.9 MB/s 2024-03-21T21:15:10.858 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T21:15:10.875 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 2024-03-21T21:15:10.934 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T21:15:10.934 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T21:15:10.935 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-21T21:15:10.935 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T21:15:10.935 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T21:15:10.935 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-21 21:15:10.823447289 +0000 2024-03-21T21:15:10.935 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-21 21:12:12.737249456 +0000 2024-03-21T21:15:10.935 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-21 21:12:12.737249456 +0000 2024-03-21T21:15:10.935 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-21 21:12:12.737249456 +0000 2024-03-21T21:15:10.935 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T21:15:11.034 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-21T21:15:11.034 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-21T21:15:11.034 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000182903 s, 2.8 MB/s 2024-03-21T21:15:11.035 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T21:15:11.051 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 2024-03-21T21:15:11.108 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T21:15:11.108 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T21:15:11.108 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-21T21:15:11.108 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T21:15:11.108 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T21:15:11.108 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-21 21:15:11.002445708 +0000 2024-03-21T21:15:11.108 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-21 21:12:13.092246342 +0000 2024-03-21T21:15:11.108 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-21 21:12:13.092246342 +0000 2024-03-21T21:15:11.108 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-21 21:12:13.092246342 +0000 2024-03-21T21:15:11.109 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T21:15:11.178 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:11 smithi066 ceph-mon[25664]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:11.180 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-21T21:15:11.181 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-21T21:15:11.181 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000191131 s, 2.7 MB/s 2024-03-21T21:15:11.182 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T21:15:11.238 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-03-21T21:15:11.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T21:15:11.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:11 smithi028 ceph-mon[24806]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:11.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:11 smithi028 ceph-mon[29568]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:12.105 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:12.013+0000 7f999e355000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:15:12.105 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:12.115+0000 7f999e355000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:15:12.834 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:12.834 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:12.835 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:12.835 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:12.835 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:12.835 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:12.835 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:12.835 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:15:12.835 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:15:12.835 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:12.836 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:12.753+0000 7f999e355000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:15:12.836 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:12.844+0000 7f999e355000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:15:12.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:12.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:12.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:12.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:12.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:12.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:12.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:12.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:15:12.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:15:12.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:15:12.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:15:12.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:13.157 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:12.930+0000 7f999e355000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:15:13.157 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:13.084+0000 7f999e355000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:15:13.157 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:13.167+0000 7f999e355000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:15:13.159 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T21:15:13.159 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T21:15:13.159 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-21T21:15:13.159 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-21T21:15:13.159 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00845457 s, 1.2 GB/s 2024-03-21T21:15:13.159 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-21T21:15:13.489 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:13.279+0000 7f999e355000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:15:13.489 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:13.366+0000 7f999e355000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:15:13.989 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:13.533+0000 7f999e355000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:15:13.989 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:13.621+0000 7f999e355000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:15:14.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:14 smithi028 ceph-mon[24806]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T21:15:14.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:14 smithi028 ceph-mon[24806]: Reconfiguring daemon mgr.y on smithi028 2024-03-21T21:15:14.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:14 smithi028 ceph-mon[29568]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T21:15:14.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:14 smithi028 ceph-mon[29568]: Reconfiguring daemon mgr.y on smithi028 2024-03-21T21:15:14.488 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:14 smithi066 ceph-mon[25664]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T21:15:14.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:14 smithi066 ceph-mon[25664]: Reconfiguring daemon mgr.y on smithi028 2024-03-21T21:15:14.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-03-21T21:15:14.990 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:14.707+0000 7f999e355000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:15:14.991 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:14.842+0000 7f999e355000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:15:15.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[24806]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:15.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:15.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:15.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:15.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:15.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[29568]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:15:15.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-mon[25664]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:15.489 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:15.037+0000 7f999e355000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:15:15.490 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:15.127+0000 7f999e355000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:15:15.490 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:15.268+0000 7f999e355000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:15:15.899 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:15.596+0000 7f999e355000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:15:15.899 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:15.823+0000 7f999e355000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:15:15.899 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:15.909+0000 7f999e355000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:15:16.238 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:16.085+0000 7f999e355000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:15:17.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:17 smithi066 ceph-mon[25664]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:17.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:15:17.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:15:17.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:17.331 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:17.034+0000 7f999e355000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:15:17.331 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:17.341+0000 7f999e355000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:15:17.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:17 smithi028 ceph-mon[29568]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:17.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:15:17.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:15:17.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:17.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:17 smithi028 ceph-mon[24806]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:17.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:15:17.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:15:17.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:17.582 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:17.438+0000 7f999e355000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:15:17.988 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:15:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:15:17.592+0000 7f999e355000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:15:18.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[29568]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:15:18.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[29568]: Standby manager daemon x started 2024-03-21T21:15:18.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:15:18.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:15:18.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:15:18.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:15:18.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[24806]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:15:18.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[24806]: Standby manager daemon x started 2024-03-21T21:15:18.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:15:18.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:15:18.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:15:18.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:18 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:15:18.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:18 smithi066 ceph-mon[25664]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:15:18.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:18 smithi066 ceph-mon[25664]: Standby manager daemon x started 2024-03-21T21:15:18.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:18 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:15:18.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:18 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:15:18.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:18 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:15:18.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:18 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/2258731086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:15:19.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[29568]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:19.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[29568]: mgrmap e14: y(active, since 65s), standbys: x 2024-03-21T21:15:19.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:15:19.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/4225891041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff470de9-e261-403a-8a5e-be3156883903"}]: dispatch 2024-03-21T21:15:19.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff470de9-e261-403a-8a5e-be3156883903"}]: dispatch 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff470de9-e261-403a-8a5e-be3156883903"}]': finished 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[29568]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[24806]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[24806]: mgrmap e14: y(active, since 65s), standbys: x 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/4225891041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff470de9-e261-403a-8a5e-be3156883903"}]: dispatch 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff470de9-e261-403a-8a5e-be3156883903"}]: dispatch 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff470de9-e261-403a-8a5e-be3156883903"}]': finished 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[24806]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T21:15:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:19 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:19.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:19 smithi066 ceph-mon[25664]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:19.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:19 smithi066 ceph-mon[25664]: mgrmap e14: y(active, since 65s), standbys: x 2024-03-21T21:15:19.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:19 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:15:19.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:19 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/4225891041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff470de9-e261-403a-8a5e-be3156883903"}]: dispatch 2024-03-21T21:15:19.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:19 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff470de9-e261-403a-8a5e-be3156883903"}]: dispatch 2024-03-21T21:15:19.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:19 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff470de9-e261-403a-8a5e-be3156883903"}]': finished 2024-03-21T21:15:19.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:19 smithi066 ceph-mon[25664]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T21:15:19.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:19 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:20.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:20 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/300060044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:15:20.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:20 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/300060044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:15:20.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:20 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/300060044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:15:21.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:21 smithi028 ceph-mon[24806]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:21.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:21 smithi028 ceph-mon[29568]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:21.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:21 smithi066 ceph-mon[25664]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:23.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:23 smithi028 ceph-mon[24806]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:23.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:23 smithi028 ceph-mon[29568]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:23.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:23 smithi066 ceph-mon[25664]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:25.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:25 smithi028 ceph-mon[24806]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:25.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:25 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T21:15:25.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:25 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:25 smithi028 ceph-mon[29568]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:25 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T21:15:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:25 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:25.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:25 smithi066 ceph-mon[25664]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:25.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:25 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T21:15:25.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:25 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:26 smithi028 ceph-mon[24806]: Deploying daemon osd.0 on smithi028 2024-03-21T21:15:26.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:26 smithi028 ceph-mon[29568]: Deploying daemon osd.0 on smithi028 2024-03-21T21:15:26.483 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:26 smithi066 ceph-mon[25664]: Deploying daemon osd.0 on smithi028 2024-03-21T21:15:27.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:27 smithi028 ceph-mon[24806]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:27.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:27 smithi028 ceph-mon[29568]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:27.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:27 smithi066 ceph-mon[25664]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:29.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:29 smithi028 ceph-mon[24806]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:29.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:29 smithi028 ceph-mon[29568]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:29.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:29 smithi066 ceph-mon[25664]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:31.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:31.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:31.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:31.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:31.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[24806]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:31.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:31.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:31.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:31.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:31.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:31 smithi028 ceph-mon[29568]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:31.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:31.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:31.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:31.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:31.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:31 smithi066 ceph-mon[25664]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:33.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:33 smithi028 ceph-mon[24806]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:33.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:33 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:33.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:33 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:33.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:33 smithi028 ceph-mon[29568]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:33.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:33 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:33.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:33 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:33.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:33 smithi066 ceph-mon[25664]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:33.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:33 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:33.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:33 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:33.908 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-03-21T21:15:34.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:34 smithi066 ceph-mon[25664]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T21:15:34.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:34 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:34.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:34 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:34.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:34 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:34.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:34 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:34.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[29568]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T21:15:34.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:34.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:34.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:34.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:34.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[24806]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T21:15:34.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:34.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:34.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:34.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:34 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:34.659 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.0.service 2024-03-21T21:15:34.663 INFO:tasks.cephadm:Deploying osd.1 on smithi028 with /dev/vg_nvme/lv_3... 2024-03-21T21:15:34.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T21:15:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[24806]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:35.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[24806]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T21:15:35.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[24806]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T21:15:35.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:35.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[24806]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:15:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[29568]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[29568]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T21:15:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[29568]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T21:15:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:35.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-mon[29568]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:15:35.437 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[32437]: 2024-03-21T21:15:35.214+0000 7f7a3ff4b700 -1 osd.0 0 waiting for initial osdmap 2024-03-21T21:15:35.437 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:15:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[32437]: 2024-03-21T21:15:35.216+0000 7f7a3a8e2700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:15:35.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:35 smithi066 ceph-mon[25664]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:35.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:35 smithi066 ceph-mon[25664]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T21:15:35.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:35 smithi066 ceph-mon[25664]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T21:15:35.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:35 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:35.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:35 smithi066 ceph-mon[25664]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:15:36.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:36 smithi066 ceph-mon[25664]: purged_snaps scrub starts 2024-03-21T21:15:36.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:36 smithi066 ceph-mon[25664]: purged_snaps scrub ok 2024-03-21T21:15:36.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:36 smithi066 ceph-mon[25664]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:15:36.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:36 smithi066 ceph-mon[25664]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T21:15:36.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:36 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:36.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:36 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:36.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[24806]: purged_snaps scrub starts 2024-03-21T21:15:36.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[24806]: purged_snaps scrub ok 2024-03-21T21:15:36.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[24806]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:15:36.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[24806]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T21:15:36.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:36.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:36.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[29568]: purged_snaps scrub starts 2024-03-21T21:15:36.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[29568]: purged_snaps scrub ok 2024-03-21T21:15:36.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[29568]: from='osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:15:36.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[29568]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T21:15:36.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:36.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:36 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:37.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:37 smithi028 ceph-mon[29568]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:37.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:37 smithi028 ceph-mon[29568]: osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836] boot 2024-03-21T21:15:37.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:37 smithi028 ceph-mon[29568]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T21:15:37.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:37 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:37.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:37 smithi028 ceph-mon[24806]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:37.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:37 smithi028 ceph-mon[24806]: osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836] boot 2024-03-21T21:15:37.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:37 smithi028 ceph-mon[24806]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T21:15:37.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:37 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:37.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:37 smithi066 ceph-mon[25664]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T21:15:37.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:37 smithi066 ceph-mon[25664]: osd.0 [v2:172.21.15.28:6802/3306214836,v1:172.21.15.28:6803/3306214836] boot 2024-03-21T21:15:37.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:37 smithi066 ceph-mon[25664]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T21:15:37.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:37 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:15:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T21:15:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T21:15:38.322 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-21T21:15:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-21T21:15:38.322 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00836195 s, 1.3 GB/s 2024-03-21T21:15:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-21T21:15:38.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:38 smithi066 ceph-mon[25664]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T21:15:38.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:38 smithi028 ceph-mon[24806]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T21:15:38.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:38 smithi028 ceph-mon[29568]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T21:15:39.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[24806]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:39.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:39.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:15:39.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:39.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:39.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[29568]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:39.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:39.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:15:39.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:39.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:39.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-03-21T21:15:39.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:39 smithi066 ceph-mon[25664]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:39.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:39.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:15:39.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:39.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:40.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:40 smithi028 ceph-mon[24806]: Detected new or changed devices on smithi028 2024-03-21T21:15:40.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:40 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi028 to 16069M 2024-03-21T21:15:40.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:40 smithi028 ceph-mon[29568]: Detected new or changed devices on smithi028 2024-03-21T21:15:40.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:40 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi028 to 16069M 2024-03-21T21:15:40.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:40 smithi066 ceph-mon[25664]: Detected new or changed devices on smithi028 2024-03-21T21:15:40.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:40 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi028 to 16069M 2024-03-21T21:15:41.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:41 smithi028 ceph-mon[24806]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:41.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:41 smithi028 ceph-mon[29568]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:41.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:41 smithi066 ceph-mon[25664]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:42.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:42 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:15:42.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:42 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:15:42.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:42 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:42.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:42 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:15:42.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:42 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:15:42.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:42 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:42.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:42 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:15:42.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:42 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:15:42.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:42 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:43.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:43 smithi028 ceph-mon[24806]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:15:43.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:43 smithi028 ceph-mon[24806]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:43.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:43 smithi028 ceph-mon[29568]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:15:43.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:43 smithi028 ceph-mon[29568]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:43.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:43 smithi066 ceph-mon[25664]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:15:43.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:43 smithi066 ceph-mon[25664]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:44.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:44 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/765448223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e259c2aa-88f8-4fcd-aba5-6323ad0e6e24"}]: dispatch 2024-03-21T21:15:44.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:44 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/765448223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e259c2aa-88f8-4fcd-aba5-6323ad0e6e24"}]': finished 2024-03-21T21:15:44.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:44 smithi066 ceph-mon[25664]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T21:15:44.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:44 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:15:44.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:44 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/765448223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e259c2aa-88f8-4fcd-aba5-6323ad0e6e24"}]: dispatch 2024-03-21T21:15:44.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:44 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/765448223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e259c2aa-88f8-4fcd-aba5-6323ad0e6e24"}]': finished 2024-03-21T21:15:44.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:44 smithi028 ceph-mon[29568]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T21:15:44.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:44 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:15:44.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:44 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/765448223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e259c2aa-88f8-4fcd-aba5-6323ad0e6e24"}]: dispatch 2024-03-21T21:15:44.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:44 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/765448223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e259c2aa-88f8-4fcd-aba5-6323ad0e6e24"}]': finished 2024-03-21T21:15:44.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:44 smithi028 ceph-mon[24806]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T21:15:44.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:44 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:15:45.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:45 smithi066 ceph-mon[25664]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:45.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:45 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2323654980' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:15:45.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:45 smithi028 ceph-mon[29568]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:45.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:45 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2323654980' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:15:45.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:45 smithi028 ceph-mon[24806]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:45.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:45 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2323654980' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:15:46.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:46 smithi066 ceph-mon[25664]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:46.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:46 smithi028 ceph-mon[24806]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:46.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:46 smithi028 ceph-mon[29568]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:49.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:49 smithi028 ceph-mon[24806]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:49.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:49 smithi028 ceph-mon[29568]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:49.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:49 smithi066 ceph-mon[25664]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:51.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:51 smithi028 ceph-mon[24806]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:51.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:51 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T21:15:51.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:51 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:51.362 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:51 smithi028 ceph-mon[29568]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:51.362 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:51 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T21:15:51.362 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:51 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:51.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:51 smithi066 ceph-mon[25664]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:51.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:51 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T21:15:51.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:51 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:52.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:52 smithi028 ceph-mon[24806]: Deploying daemon osd.1 on smithi028 2024-03-21T21:15:52.341 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:52 smithi028 ceph-mon[29568]: Deploying daemon osd.1 on smithi028 2024-03-21T21:15:52.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:52 smithi066 ceph-mon[25664]: Deploying daemon osd.1 on smithi028 2024-03-21T21:15:53.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:53 smithi028 ceph-mon[29568]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:53.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:53 smithi028 ceph-mon[24806]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:53.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:53 smithi066 ceph-mon[25664]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:55 smithi028 ceph-mon[24806]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:55.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:55 smithi028 ceph-mon[29568]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:55.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:55 smithi066 ceph-mon[25664]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:56.225 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:56 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:56.225 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:56 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:56.226 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:56 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:56.226 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:56 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:56.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:56 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:56.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:56 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:56.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:56 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:56.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:56 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:56.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:56 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:15:56.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:56 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:56.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:56 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:15:56.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:56 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:15:57.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:57 smithi028 ceph-mon[24806]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:57.412 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:57 smithi028 ceph-mon[29568]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:57.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:57 smithi066 ceph-mon[25664]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:59.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:59 smithi028 ceph-mon[24806]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:59.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:59 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:59.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:59 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:59.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:15:59 smithi028 ceph-mon[24806]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T21:15:59.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:59 smithi028 ceph-mon[29568]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:59.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:59 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:59.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:59 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:59.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:15:59 smithi028 ceph-mon[29568]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T21:15:59.423 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 1 on host 'smithi028' 2024-03-21T21:15:59.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:59 smithi066 ceph-mon[25664]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:15:59.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:59 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:59.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:59 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:15:59.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:15:59 smithi066 ceph-mon[25664]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T21:16:00.087 DEBUG:teuthology.orchestra.run.smithi028:osd.1> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.1.service 2024-03-21T21:16:00.089 INFO:tasks.cephadm:Deploying osd.2 on smithi028 with /dev/vg_nvme/lv_2... 2024-03-21T21:16:00.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T21:16:00.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:00.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:00.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[29568]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[29568]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[29568]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[29568]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:00.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[24806]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T21:16:00.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[24806]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T21:16:00.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:00.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[24806]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:16:00.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-mon[24806]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:16:00.688 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[35393]: 2024-03-21T21:16:00.541+0000 7f4d4c0fd700 -1 osd.1 0 waiting for initial osdmap 2024-03-21T21:16:00.688 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:16:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[35393]: 2024-03-21T21:16:00.544+0000 7f4d46a94700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:16:00.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:00 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:00.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:00 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:00.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:00 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:00.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:00 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:00.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:00 smithi066 ceph-mon[25664]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T21:16:00.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:00 smithi066 ceph-mon[25664]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T21:16:00.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:00 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:00.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:00 smithi066 ceph-mon[25664]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:16:00.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:00 smithi066 ceph-mon[25664]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:16:01.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:01 smithi028 ceph-mon[24806]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:16:01.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:01 smithi028 ceph-mon[24806]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T21:16:01.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:01 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:01.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:01 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:01.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:01 smithi028 ceph-mon[29568]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:16:01.891 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:01 smithi028 ceph-mon[29568]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T21:16:01.891 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:01 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:01.891 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:01 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:01.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:01 smithi066 ceph-mon[25664]: from='osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:16:01.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:01 smithi066 ceph-mon[25664]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T21:16:01.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:01 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:01.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:01 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:02.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[29568]: purged_snaps scrub starts 2024-03-21T21:16:02.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[29568]: purged_snaps scrub ok 2024-03-21T21:16:02.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:02.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[29568]: osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520] boot 2024-03-21T21:16:02.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[29568]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T21:16:02.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:02.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[29568]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:16:02.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[24806]: purged_snaps scrub starts 2024-03-21T21:16:02.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[24806]: purged_snaps scrub ok 2024-03-21T21:16:02.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:02.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[24806]: osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520] boot 2024-03-21T21:16:02.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[24806]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T21:16:02.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:02.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:02 smithi028 ceph-mon[24806]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:16:02.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:02 smithi066 ceph-mon[25664]: purged_snaps scrub starts 2024-03-21T21:16:02.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:02 smithi066 ceph-mon[25664]: purged_snaps scrub ok 2024-03-21T21:16:02.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:02 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:02.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:02 smithi066 ceph-mon[25664]: osd.1 [v2:172.21.15.28:6810/1005734520,v1:172.21.15.28:6811/1005734520] boot 2024-03-21T21:16:02.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:02 smithi066 ceph-mon[25664]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T21:16:02.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:02 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:16:02.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:02 smithi066 ceph-mon[25664]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T21:16:03.299 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T21:16:03.299 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T21:16:03.300 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-21T21:16:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-21T21:16:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00943756 s, 1.1 GB/s 2024-03-21T21:16:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-21T21:16:03.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:03 smithi028 ceph-mon[29568]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T21:16:03.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:03 smithi028 ceph-mon[24806]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T21:16:03.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:03 smithi066 ceph-mon[25664]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T21:16:04.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-03-21T21:16:04.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:04 smithi028 ceph-mon[24806]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:04.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:04 smithi028 ceph-mon[29568]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:04.989 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:04 smithi066 ceph-mon[25664]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:06.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[29568]: Detected new or changed devices on smithi028 2024-03-21T21:16:06.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:06.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:06.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:06.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi028 to 8034M 2024-03-21T21:16:06.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:06.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:06.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[24806]: Detected new or changed devices on smithi028 2024-03-21T21:16:06.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:06.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:06.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:06.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi028 to 8034M 2024-03-21T21:16:06.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:06.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:05 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:06.239 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:05 smithi066 ceph-mon[25664]: Detected new or changed devices on smithi028 2024-03-21T21:16:06.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:05 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:06.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:05 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:06.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:05 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:06.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:05 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi028 to 8034M 2024-03-21T21:16:06.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:05 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:06.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:05 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:06.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:06 smithi028 ceph-mon[24806]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:06.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:06 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:16:06.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:06 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:16:06.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:06 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:06.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:06 smithi028 ceph-mon[29568]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:06.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:06 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:16:06.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:06 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:16:06.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:06 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:07.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:06 smithi066 ceph-mon[25664]: pgmap v60: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:07.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:06 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:16:07.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:06 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:16:07.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:06 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:07.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:07 smithi028 ceph-mon[24806]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:16:07.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:07 smithi028 ceph-mon[29568]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:16:08.240 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:07 smithi066 ceph-mon[25664]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:16:09.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[24806]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:09.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3973407788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6d94c3-b3f1-4274-a83c-7e8a1fa291aa"}]: dispatch 2024-03-21T21:16:09.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3973407788' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b6d94c3-b3f1-4274-a83c-7e8a1fa291aa"}]': finished 2024-03-21T21:16:09.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[24806]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T21:16:09.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:09.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[29568]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:09.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3973407788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6d94c3-b3f1-4274-a83c-7e8a1fa291aa"}]: dispatch 2024-03-21T21:16:09.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3973407788' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b6d94c3-b3f1-4274-a83c-7e8a1fa291aa"}]': finished 2024-03-21T21:16:09.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[29568]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T21:16:09.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:09 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:09.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:09 smithi066 ceph-mon[25664]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:09.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:09 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3973407788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6d94c3-b3f1-4274-a83c-7e8a1fa291aa"}]: dispatch 2024-03-21T21:16:09.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:09 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3973407788' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b6d94c3-b3f1-4274-a83c-7e8a1fa291aa"}]': finished 2024-03-21T21:16:09.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:09 smithi066 ceph-mon[25664]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T21:16:09.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:09 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:10.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:10 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1392739690' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:16:10.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:10 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1392739690' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:16:10.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:10 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1392739690' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:16:11.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:11 smithi028 ceph-mon[24806]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:11.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:11 smithi028 ceph-mon[29568]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:11.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:11 smithi066 ceph-mon[25664]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:13.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:13 smithi028 ceph-mon[24806]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:13.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:13 smithi028 ceph-mon[29568]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:13.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:13 smithi066 ceph-mon[25664]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:15.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:15 smithi028 ceph-mon[24806]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:15.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:16:15.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:15 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:16:15.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:15 smithi028 ceph-mon[29568]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:15.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:16:15.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:15 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:16:15.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:15 smithi066 ceph-mon[25664]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:15.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:16:15.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:15 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:16:16.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:16 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T21:16:16.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:16 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:16.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:16 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T21:16:16.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:16 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:16.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:16 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T21:16:16.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:16 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:17.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:17 smithi028 ceph-mon[24806]: Deploying daemon osd.2 on smithi028 2024-03-21T21:16:17.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:17 smithi028 ceph-mon[24806]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:17.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:17 smithi028 ceph-mon[29568]: Deploying daemon osd.2 on smithi028 2024-03-21T21:16:17.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:17 smithi028 ceph-mon[29568]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:17.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:17 smithi066 ceph-mon[25664]: Deploying daemon osd.2 on smithi028 2024-03-21T21:16:17.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:17 smithi066 ceph-mon[25664]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:19.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:19 smithi028 ceph-mon[24806]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:19.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:19 smithi028 ceph-mon[29568]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:19.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:19 smithi066 ceph-mon[25664]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:21.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[24806]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:21.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:21.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:21.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:21.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:21.229 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[29568]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:21.229 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:21.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:21.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:21.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:21 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:21.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:21 smithi066 ceph-mon[25664]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:21.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:21 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:21.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:21 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:21.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:21 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:21.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:21 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:23.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:23 smithi028 ceph-mon[29568]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:23.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:23 smithi028 ceph-mon[24806]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:23.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:23 smithi066 ceph-mon[25664]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:23.862 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 2 on host 'smithi028' 2024-03-21T21:16:24.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:24 smithi028 ceph-mon[24806]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T21:16:24.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:24 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:24.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:24 smithi028 ceph-mon[29568]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T21:16:24.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:24 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:24.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:24 smithi066 ceph-mon[25664]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T21:16:24.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:24 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:24.500 DEBUG:teuthology.orchestra.run.smithi028:osd.2> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.2.service 2024-03-21T21:16:24.504 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_1... 2024-03-21T21:16:24.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T21:16:25.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[24806]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:25.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[24806]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T21:16:25.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[24806]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T21:16:25.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:25.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[24806]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:16:25.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[29568]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[29568]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T21:16:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[29568]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T21:16:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[29568]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:16:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:25.438 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[38316]: 2024-03-21T21:16:25.192+0000 7fc381c79700 -1 osd.2 0 waiting for initial osdmap 2024-03-21T21:16:25.438 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:16:25 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[38316]: 2024-03-21T21:16:25.196+0000 7fc37d612700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:16:25.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:25 smithi066 ceph-mon[25664]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:25.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:25 smithi066 ceph-mon[25664]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T21:16:25.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:25 smithi066 ceph-mon[25664]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T21:16:25.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:25 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:25.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:25 smithi066 ceph-mon[25664]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:16:25.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:25 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:26.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:26 smithi028 ceph-mon[24806]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:16:26.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:26 smithi028 ceph-mon[24806]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T21:16:26.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:26.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:26.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:26 smithi028 ceph-mon[29568]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:16:26.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:26 smithi028 ceph-mon[29568]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T21:16:26.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:26.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:26.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:26 smithi066 ceph-mon[25664]: from='osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:16:26.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:26 smithi066 ceph-mon[25664]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T21:16:26.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:26.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:26.931 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T21:16:26.931 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T21:16:26.931 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-21T21:16:26.931 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-21T21:16:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00975915 s, 1.1 GB/s 2024-03-21T21:16:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-21T21:16:27.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[29568]: purged_snaps scrub starts 2024-03-21T21:16:27.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[29568]: purged_snaps scrub ok 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[29568]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[29568]: osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792] boot 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[29568]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[24806]: purged_snaps scrub starts 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[24806]: purged_snaps scrub ok 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[24806]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[24806]: osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792] boot 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[24806]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T21:16:27.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:27 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-21T21:16:27.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:27 smithi066 ceph-mon[25664]: purged_snaps scrub starts 2024-03-21T21:16:27.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:27 smithi066 ceph-mon[25664]: purged_snaps scrub ok 2024-03-21T21:16:27.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:27 smithi066 ceph-mon[25664]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T21:16:27.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:27 smithi066 ceph-mon[25664]: osd.2 [v2:172.21.15.28:6818/1632373792,v1:172.21.15.28:6819/1632373792] boot 2024-03-21T21:16:27.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:27 smithi066 ceph-mon[25664]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T21:16:27.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:27 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:16:27.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:27 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-21T21:16:28.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:28 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-21T21:16:28.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:28 smithi066 ceph-mon[25664]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T21:16:28.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:28 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T21:16:28.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:28 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-21T21:16:28.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:28 smithi028 ceph-mon[29568]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T21:16:28.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:28 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T21:16:28.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:28 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-21T21:16:28.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:28 smithi028 ceph-mon[24806]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T21:16:28.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:28 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T21:16:28.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-03-21T21:16:29.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:29 smithi066 ceph-mon[25664]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:29.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:29 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T21:16:29.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:29 smithi066 ceph-mon[25664]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T21:16:29.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:29 smithi028 ceph-mon[29568]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:29.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:29 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T21:16:29.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:29 smithi028 ceph-mon[29568]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T21:16:29.498 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:16:29 smithi028 sudo[39605]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T21:16:29.498 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:16:29 smithi028 sudo[39605]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:29.498 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:16:29 smithi028 sudo[39605]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:29.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:29 smithi028 ceph-mon[24806]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:29.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:29 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T21:16:29.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:29 smithi028 ceph-mon[24806]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T21:16:29.775 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:16:29 smithi028 sudo[39609]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T21:16:29.775 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:16:29 smithi028 sudo[39609]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:29.775 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:16:29 smithi028 sudo[39609]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:30.046 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:16:29 smithi028 sudo[39612]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T21:16:30.046 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:16:29 smithi028 sudo[39612]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:30.046 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:16:29 smithi028 sudo[39612]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:30.303 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39620]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T21:16:30.303 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39620]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:30.303 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39620]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:30.304 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:30.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T21:16:30.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:30.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:30 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:30.574 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39630]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T21:16:30.575 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39630]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:30.575 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39630]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:30.830 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39650]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T21:16:30.830 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39650]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:30.830 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39650]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:31.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39675]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T21:16:31.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39675]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:31.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:30 smithi028 sudo[39675]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:31.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 sudo[39679]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T21:16:31.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 sudo[39679]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:31.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 sudo[39679]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:31.621 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 sudo[39740]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T21:16:31.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 sudo[39740]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:31.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 sudo[39740]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:31.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: Detected new or changed devices on smithi028 2024-03-21T21:16:31.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi028 to 5356M 2024-03-21T21:16:31.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:31.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T21:16:31.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T21:16:31.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:16:31.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:16:31.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:16:31.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 sudo[39769]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 sudo[39769]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 sudo[39769]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: Detected new or changed devices on smithi028 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi028 to 5356M 2024-03-21T21:16:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:16:31.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:16:31.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:31.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: Detected new or changed devices on smithi028 2024-03-21T21:16:31.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi028 to 5356M 2024-03-21T21:16:31.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:31.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T21:16:31.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:16:31.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:32.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 sudo[27191]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T21:16:32.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 sudo[27191]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:32.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 sudo[27191]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T21:16:32.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T21:16:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:32 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:32.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 sudo[27194]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T21:16:32.990 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 sudo[27194]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 sudo[27194]: pam_unix(sudo:session): session closed for user root 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T21:16:32.991 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:32 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:34.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2827035691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1bfc46b-7ec2-46e0-88e5-f417d2a01db7"}]: dispatch 2024-03-21T21:16:34.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1bfc46b-7ec2-46e0-88e5-f417d2a01db7"}]: dispatch 2024-03-21T21:16:34.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1bfc46b-7ec2-46e0-88e5-f417d2a01db7"}]': finished 2024-03-21T21:16:34.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[24806]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T21:16:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3345042283' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:16:34.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2827035691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1bfc46b-7ec2-46e0-88e5-f417d2a01db7"}]: dispatch 2024-03-21T21:16:34.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1bfc46b-7ec2-46e0-88e5-f417d2a01db7"}]: dispatch 2024-03-21T21:16:34.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1bfc46b-7ec2-46e0-88e5-f417d2a01db7"}]': finished 2024-03-21T21:16:34.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[29568]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T21:16:34.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:34.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:34 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3345042283' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:16:34.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:34 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2827035691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1bfc46b-7ec2-46e0-88e5-f417d2a01db7"}]: dispatch 2024-03-21T21:16:34.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:34 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1bfc46b-7ec2-46e0-88e5-f417d2a01db7"}]: dispatch 2024-03-21T21:16:34.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:34 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1bfc46b-7ec2-46e0-88e5-f417d2a01db7"}]': finished 2024-03-21T21:16:34.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:34 smithi066 ceph-mon[25664]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T21:16:34.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:34 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:34.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:34 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3345042283' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:16:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:35 smithi028 ceph-mon[29568]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:35.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:35 smithi028 ceph-mon[24806]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:35.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:35 smithi066 ceph-mon[25664]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:37.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:37 smithi028 ceph-mon[24806]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:37.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:37 smithi028 ceph-mon[29568]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:37.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:37 smithi066 ceph-mon[25664]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:39.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:39 smithi028 ceph-mon[24806]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 170 KiB/s wr, 12 op/s 2024-03-21T21:16:39.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:39 smithi028 ceph-mon[29568]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 170 KiB/s wr, 12 op/s 2024-03-21T21:16:39.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:39 smithi066 ceph-mon[25664]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 170 KiB/s wr, 12 op/s 2024-03-21T21:16:40.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:40 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T21:16:40.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:40 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:40.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:40 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T21:16:40.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:40 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:40.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:40 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T21:16:40.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:40 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:41.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:41 smithi028 ceph-mon[24806]: Deploying daemon osd.3 on smithi028 2024-03-21T21:16:41.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:41 smithi028 ceph-mon[24806]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 118 KiB/s wr, 8 op/s 2024-03-21T21:16:41.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:41 smithi028 ceph-mon[29568]: Deploying daemon osd.3 on smithi028 2024-03-21T21:16:41.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:41 smithi028 ceph-mon[29568]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 118 KiB/s wr, 8 op/s 2024-03-21T21:16:41.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:41 smithi066 ceph-mon[25664]: Deploying daemon osd.3 on smithi028 2024-03-21T21:16:41.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:41 smithi066 ceph-mon[25664]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 118 KiB/s wr, 8 op/s 2024-03-21T21:16:43.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:43 smithi028 ceph-mon[24806]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 91 KiB/s wr, 6 op/s 2024-03-21T21:16:43.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:43 smithi028 ceph-mon[29568]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 91 KiB/s wr, 6 op/s 2024-03-21T21:16:43.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:43 smithi066 ceph-mon[25664]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 91 KiB/s wr, 6 op/s 2024-03-21T21:16:45.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:45 smithi028 ceph-mon[24806]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-03-21T21:16:45.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:45 smithi028 ceph-mon[29568]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-03-21T21:16:45.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:45 smithi066 ceph-mon[25664]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-03-21T21:16:46.740 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:46 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:46.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:46 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:46.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:46 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:46.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:46 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:46.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:46 smithi066 ceph-mon[25664]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T21:16:46.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:46.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:46.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:46.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:46.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[29568]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T21:16:46.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:46.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:46.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:46.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:46.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:46 smithi028 ceph-mon[24806]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T21:16:48.521 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-03-21T21:16:49.219 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.3.service 2024-03-21T21:16:49.221 INFO:tasks.cephadm:Deploying osd.4 on smithi066 with /dev/vg_nvme/lv_4... 2024-03-21T21:16:49.222 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T21:16:49.457 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:49 smithi028 ceph-mon[29568]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T21:16:49.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:49 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:49.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:49 smithi028 ceph-mon[29568]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T21:16:49.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:49 smithi028 ceph-mon[24806]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T21:16:49.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:49 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:49.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:49 smithi028 ceph-mon[24806]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T21:16:49.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:49 smithi066 ceph-mon[25664]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T21:16:49.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:49 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:49.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:49 smithi066 ceph-mon[25664]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T21:16:50.479 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:50 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:50.479 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:50 smithi066 ceph-mon[25664]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T21:16:50.479 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:50 smithi066 ceph-mon[25664]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T21:16:50.479 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:50 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:50.479 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:50 smithi066 ceph-mon[25664]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:16:50.479 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:50 smithi066 ceph-mon[25664]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:50.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:50.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[29568]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T21:16:50.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[29568]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T21:16:50.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:50.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[29568]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:16:50.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[29568]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:50.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:50.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[24806]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T21:16:50.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[24806]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T21:16:50.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:50.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[24806]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:16:50.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-mon[24806]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:50.688 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[41298]: 2024-03-21T21:16:50.535+0000 7f6d3b7e5700 -1 osd.3 0 waiting for initial osdmap 2024-03-21T21:16:50.688 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:16:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[41298]: 2024-03-21T21:16:50.539+0000 7f6d34979700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:16:51.217 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T21:16:51.217 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T21:16:51.217 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-03-21T21:16:51.218 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-03-21T21:16:51.218 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114272 s, 918 MB/s 2024-03-21T21:16:51.218 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-03-21T21:16:51.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:51 smithi028 ceph-mon[24806]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:16:51.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:51 smithi028 ceph-mon[24806]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T21:16:51.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:51 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:51.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:51 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:51.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:51 smithi028 ceph-mon[29568]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:16:51.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:51 smithi028 ceph-mon[29568]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T21:16:51.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:51 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:51.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:51 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:51.801 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:51 smithi066 ceph-mon[25664]: from='osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-21T21:16:51.801 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:51 smithi066 ceph-mon[25664]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T21:16:51.801 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:51 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:51.802 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:51 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:51.824 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_4 2024-03-21T21:16:52.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:52 smithi066 ceph-mon[25664]: purged_snaps scrub starts 2024-03-21T21:16:52.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:52 smithi066 ceph-mon[25664]: purged_snaps scrub ok 2024-03-21T21:16:52.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:52 smithi066 ceph-mon[25664]: osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441] boot 2024-03-21T21:16:52.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:52 smithi066 ceph-mon[25664]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T21:16:52.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:52 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:52.734 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:52 smithi066 ceph-mon[25664]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:52.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[29568]: purged_snaps scrub starts 2024-03-21T21:16:52.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[29568]: purged_snaps scrub ok 2024-03-21T21:16:52.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[29568]: osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441] boot 2024-03-21T21:16:52.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[29568]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T21:16:52.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:52.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[29568]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:52.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[24806]: purged_snaps scrub starts 2024-03-21T21:16:52.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[24806]: purged_snaps scrub ok 2024-03-21T21:16:52.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[24806]: osd.3 [v2:172.21.15.28:6826/352039441,v1:172.21.15.28:6827/352039441] boot 2024-03-21T21:16:52.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[24806]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T21:16:52.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:16:52.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:52 smithi028 ceph-mon[24806]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T21:16:54.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: Detected new or changed devices on smithi028 2024-03-21T21:16:54.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi028 to 4017M 2024-03-21T21:16:54.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: Detected new or changed devices on smithi028 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi028 to 4017M 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:16:54.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:53 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: Detected new or changed devices on smithi028 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi028 to 4017M 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:16:54.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:54.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:16:54.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:16:54.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T21:16:54.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:16:54.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:16:54.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:53 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:16:55.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:54 smithi028 ceph-mon[24806]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:16:55.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:54 smithi028 ceph-mon[24806]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:16:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:54 smithi028 ceph-mon[29568]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:16:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:54 smithi028 ceph-mon[29568]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:16:55.218 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:54 smithi066 ceph-mon[25664]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:16:55.218 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:54 smithi066 ceph-mon[25664]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:16:57.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:57 smithi066 ceph-mon[25664]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:16:57.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:57 smithi066 ceph-mon[25664]: from='client.? 172.21.15.66:0/2094935790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4efce96-eefe-4ed7-b490-dcef651634cf"}]: dispatch 2024-03-21T21:16:57.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:57 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4efce96-eefe-4ed7-b490-dcef651634cf"}]: dispatch 2024-03-21T21:16:57.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:57 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4efce96-eefe-4ed7-b490-dcef651634cf"}]': finished 2024-03-21T21:16:57.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:57 smithi066 ceph-mon[25664]: osdmap e27: 5 total, 4 up, 5 in 2024-03-21T21:16:57.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:57 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:16:57.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:57 smithi066 ceph-mon[25664]: from='client.? 172.21.15.66:0/605198837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:16:57.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[24806]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:16:57.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/2094935790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4efce96-eefe-4ed7-b490-dcef651634cf"}]: dispatch 2024-03-21T21:16:57.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4efce96-eefe-4ed7-b490-dcef651634cf"}]: dispatch 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4efce96-eefe-4ed7-b490-dcef651634cf"}]': finished 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[24806]: osdmap e27: 5 total, 4 up, 5 in 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/605198837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[29568]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/2094935790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4efce96-eefe-4ed7-b490-dcef651634cf"}]: dispatch 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4efce96-eefe-4ed7-b490-dcef651634cf"}]: dispatch 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4efce96-eefe-4ed7-b490-dcef651634cf"}]': finished 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[29568]: osdmap e27: 5 total, 4 up, 5 in 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:16:57.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:57 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/605198837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:16:59.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:16:59 smithi066 ceph-mon[25664]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:16:59.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:16:59 smithi028 ceph-mon[24806]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:16:59.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:16:59 smithi028 ceph-mon[29568]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:01.377 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:01 smithi066 ceph-mon[25664]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:01.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:01 smithi028 ceph-mon[24806]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:01.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:01 smithi028 ceph-mon[29568]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:03.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:03 smithi066 ceph-mon[25664]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:03.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:03 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T21:17:03.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:03 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:03.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:03 smithi028 ceph-mon[24806]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:03.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:03 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T21:17:03.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:03 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:03.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:03 smithi028 ceph-mon[29568]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:03.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:03 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T21:17:03.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:03 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:04.339 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:04 smithi066 ceph-mon[25664]: Deploying daemon osd.4 on smithi066 2024-03-21T21:17:04.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:04 smithi028 ceph-mon[24806]: Deploying daemon osd.4 on smithi066 2024-03-21T21:17:04.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:04 smithi028 ceph-mon[29568]: Deploying daemon osd.4 on smithi066 2024-03-21T21:17:05.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:05 smithi066 ceph-mon[25664]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:05.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:05 smithi028 ceph-mon[24806]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:05.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:05 smithi028 ceph-mon[29568]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:07.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:07 smithi066 ceph-mon[25664]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:07.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:07 smithi028 ceph-mon[24806]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:07.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:07 smithi028 ceph-mon[29568]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:08.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:08 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:08.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:08 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:08.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:08 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:08.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:08 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:08.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:08 smithi066 ceph-mon[25664]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:08.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:08.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:08.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:08.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:08.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[24806]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:08.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:08.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:08.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:08.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:08.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:08 smithi028 ceph-mon[29568]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:10.214 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 4 on host 'smithi066' 2024-03-21T21:17:11.160 DEBUG:teuthology.orchestra.run.smithi066:osd.4> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.4.service 2024-03-21T21:17:11.163 INFO:tasks.cephadm:Deploying osd.5 on smithi066 with /dev/vg_nvme/lv_3... 2024-03-21T21:17:11.163 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T21:17:11.395 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:11 smithi066 ceph-mon[25664]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:11.395 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:11 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:11.396 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:11 smithi066 ceph-mon[25664]: from='osd.4 [v2:172.21.15.66:6800/1921306455,v1:172.21.15.66:6801/1921306455]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:17:11.396 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:11 smithi066 ceph-mon[25664]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:17:11.396 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:11 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:11.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[24806]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:11.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:11.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[24806]: from='osd.4 [v2:172.21.15.66:6800/1921306455,v1:172.21.15.66:6801/1921306455]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:17:11.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[24806]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:17:11.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:11.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[29568]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:11.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:11.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[29568]: from='osd.4 [v2:172.21.15.66:6800/1921306455,v1:172.21.15.66:6801/1921306455]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:17:11.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[29568]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:17:11.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:11 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:12.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:12 smithi066 ceph-mon[25664]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T21:17:12.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:12 smithi066 ceph-mon[25664]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T21:17:12.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:12 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:12.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:12 smithi066 ceph-mon[25664]: from='osd.4 [v2:172.21.15.66:6800/1921306455,v1:172.21.15.66:6801/1921306455]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:12.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:12 smithi066 ceph-mon[25664]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:12.492 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:17:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[29040]: 2024-03-21T21:17:12.237+0000 7fca63448700 -1 osd.4 0 waiting for initial osdmap 2024-03-21T21:17:12.492 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:17:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[29040]: 2024-03-21T21:17:12.242+0000 7fca5e5e0700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:17:12.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[24806]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T21:17:12.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[24806]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T21:17:12.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:12.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[24806]: from='osd.4 [v2:172.21.15.66:6800/1921306455,v1:172.21.15.66:6801/1921306455]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:12.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[24806]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:12.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[29568]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T21:17:12.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[29568]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T21:17:12.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:12.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[29568]: from='osd.4 [v2:172.21.15.66:6800/1921306455,v1:172.21.15.66:6801/1921306455]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:12.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:12 smithi028 ceph-mon[29568]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:13.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:13 smithi066 ceph-mon[25664]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:13.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:13 smithi066 ceph-mon[25664]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:17:13.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:13 smithi066 ceph-mon[25664]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T21:17:13.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:13 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:13.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:13 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:13.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:13 smithi066 ceph-mon[25664]: osd.4 [v2:172.21.15.66:6800/1921306455,v1:172.21.15.66:6801/1921306455] boot 2024-03-21T21:17:13.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:13 smithi066 ceph-mon[25664]: osdmap e30: 5 total, 5 up, 5 in 2024-03-21T21:17:13.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:13 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[24806]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[24806]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[24806]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[24806]: osd.4 [v2:172.21.15.66:6800/1921306455,v1:172.21.15.66:6801/1921306455] boot 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[24806]: osdmap e30: 5 total, 5 up, 5 in 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[29568]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[29568]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[29568]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:13.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[29568]: osd.4 [v2:172.21.15.66:6800/1921306455,v1:172.21.15.66:6801/1921306455] boot 2024-03-21T21:17:13.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[29568]: osdmap e30: 5 total, 5 up, 5 in 2024-03-21T21:17:13.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:13 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:17:13.746 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T21:17:13.747 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T21:17:13.747 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-03-21T21:17:13.747 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-03-21T21:17:13.747 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136789 s, 767 MB/s 2024-03-21T21:17:13.747 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-03-21T21:17:14.422 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:14 smithi066 ceph-mon[25664]: purged_snaps scrub starts 2024-03-21T21:17:14.422 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:14 smithi066 ceph-mon[25664]: purged_snaps scrub ok 2024-03-21T21:17:14.422 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:14 smithi066 ceph-mon[25664]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T21:17:14.422 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:14 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:17:14.422 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:14 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:17:14.446 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_3 2024-03-21T21:17:14.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[24806]: purged_snaps scrub starts 2024-03-21T21:17:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[24806]: purged_snaps scrub ok 2024-03-21T21:17:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[24806]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T21:17:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:17:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:17:14.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[29568]: purged_snaps scrub starts 2024-03-21T21:17:14.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[29568]: purged_snaps scrub ok 2024-03-21T21:17:14.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[29568]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T21:17:14.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:17:14.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:14 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:17:15.353 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:15 smithi066 ceph-mon[25664]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:15.353 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:15 smithi066 ceph-mon[25664]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T21:17:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:15 smithi028 ceph-mon[24806]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:15 smithi028 ceph-mon[24806]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T21:17:15.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:15 smithi028 ceph-mon[29568]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:15.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:15 smithi028 ceph-mon[29568]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T21:17:17.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: Detected new or changed devices on smithi066 2024-03-21T21:17:17.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:17:17.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi066 to 17088M 2024-03-21T21:17:17.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:17.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:17.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[24806]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: Detected new or changed devices on smithi066 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi066 to 17088M 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:17.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:17.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:17.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:17 smithi028 ceph-mon[29568]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T21:17:17.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: Detected new or changed devices on smithi066 2024-03-21T21:17:17.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:17:17.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi066 to 17088M 2024-03-21T21:17:17.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:17.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:17.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:17.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:17.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:17.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:17 smithi066 ceph-mon[25664]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T21:17:18.067 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:18 smithi066 ceph-mon[25664]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:17:18.067 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:18 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:17:18.067 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:18 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:17:18.067 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:18 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:18.067 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:18 smithi066 ceph-mon[25664]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T21:17:18.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[24806]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:17:18.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:17:18.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:17:18.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:18.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[24806]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T21:17:18.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[29568]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:17:18.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:17:18.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:17:18.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:18.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:18 smithi028 ceph-mon[29568]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T21:17:19.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:19 smithi028 ceph-mon[24806]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T21:17:19.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:19 smithi028 ceph-mon[29568]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T21:17:19.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:19 smithi066 ceph-mon[25664]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T21:17:20.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/4204712046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54c2da13-75c9-4c8a-93f6-d34110ac4cfe"}]: dispatch 2024-03-21T21:17:20.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54c2da13-75c9-4c8a-93f6-d34110ac4cfe"}]: dispatch 2024-03-21T21:17:20.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54c2da13-75c9-4c8a-93f6-d34110ac4cfe"}]': finished 2024-03-21T21:17:20.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[24806]: osdmap e35: 6 total, 5 up, 6 in 2024-03-21T21:17:20.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:20.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/3104522299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:17:20.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/4204712046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54c2da13-75c9-4c8a-93f6-d34110ac4cfe"}]: dispatch 2024-03-21T21:17:20.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54c2da13-75c9-4c8a-93f6-d34110ac4cfe"}]: dispatch 2024-03-21T21:17:20.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54c2da13-75c9-4c8a-93f6-d34110ac4cfe"}]': finished 2024-03-21T21:17:20.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[29568]: osdmap e35: 6 total, 5 up, 6 in 2024-03-21T21:17:20.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:20.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:20 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/3104522299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:17:20.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:20 smithi066 ceph-mon[25664]: from='client.? 172.21.15.66:0/4204712046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54c2da13-75c9-4c8a-93f6-d34110ac4cfe"}]: dispatch 2024-03-21T21:17:20.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:20 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54c2da13-75c9-4c8a-93f6-d34110ac4cfe"}]: dispatch 2024-03-21T21:17:20.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:20 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54c2da13-75c9-4c8a-93f6-d34110ac4cfe"}]': finished 2024-03-21T21:17:20.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:20 smithi066 ceph-mon[25664]: osdmap e35: 6 total, 5 up, 6 in 2024-03-21T21:17:20.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:20 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:20.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:20 smithi066 ceph-mon[25664]: from='client.? 172.21.15.66:0/3104522299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:17:21.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:21 smithi028 ceph-mon[24806]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T21:17:21.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:21 smithi028 ceph-mon[29568]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T21:17:21.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:21 smithi066 ceph-mon[25664]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T21:17:23.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:23 smithi028 ceph-mon[29568]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T21:17:23.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:23 smithi028 ceph-mon[24806]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T21:17:23.445 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:23 smithi066 ceph-mon[25664]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T21:17:25.424 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:25 smithi066 ceph-mon[25664]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T21:17:25.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:25 smithi028 ceph-mon[24806]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T21:17:25.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:25 smithi028 ceph-mon[29568]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T21:17:26.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T21:17:26.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:26.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T21:17:26.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:26.453 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T21:17:26.453 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:27.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:27 smithi066 ceph-mon[25664]: Deploying daemon osd.5 on smithi066 2024-03-21T21:17:27.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:27 smithi066 ceph-mon[25664]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T21:17:27.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:27 smithi028 ceph-mon[24806]: Deploying daemon osd.5 on smithi066 2024-03-21T21:17:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:27 smithi028 ceph-mon[24806]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T21:17:27.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:27 smithi028 ceph-mon[29568]: Deploying daemon osd.5 on smithi066 2024-03-21T21:17:27.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:27 smithi028 ceph-mon[29568]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T21:17:29.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:29 smithi028 ceph-mon[24806]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:29.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:29 smithi028 ceph-mon[29568]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:29.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:29 smithi066 ceph-mon[25664]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:31.376 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:31 smithi066 ceph-mon[25664]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:31.376 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:31.376 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:31.376 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:31.376 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:31 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:31.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[29568]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:31.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:31.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:31.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:31.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:31.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[24806]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:31.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:31.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:31.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:31.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:31 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:33.125 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 5 on host 'smithi066' 2024-03-21T21:17:33.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:33 smithi066 ceph-mon[25664]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:33.241 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:33 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:33.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:33 smithi028 ceph-mon[29568]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:33.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:33 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:33.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:33 smithi028 ceph-mon[24806]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:33.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:33 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:33.883 DEBUG:teuthology.orchestra.run.smithi066:osd.5> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.5.service 2024-03-21T21:17:33.887 INFO:tasks.cephadm:Deploying osd.6 on smithi066 with /dev/vg_nvme/lv_2... 2024-03-21T21:17:33.887 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T21:17:34.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:34 smithi028 ceph-mon[29568]: from='osd.5 [v2:172.21.15.66:6808/174977721,v1:172.21.15.66:6809/174977721]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:17:34.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:34 smithi028 ceph-mon[29568]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:17:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:34 smithi028 ceph-mon[24806]: from='osd.5 [v2:172.21.15.66:6808/174977721,v1:172.21.15.66:6809/174977721]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:17:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:34 smithi028 ceph-mon[24806]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:17:34.458 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:34 smithi066 ceph-mon[25664]: from='osd.5 [v2:172.21.15.66:6808/174977721,v1:172.21.15.66:6809/174977721]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:17:34.458 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:34 smithi066 ceph-mon[25664]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:17:35.455 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:17:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[31986]: 2024-03-21T21:17:35.204+0000 7fb363bec700 -1 osd.5 0 waiting for initial osdmap 2024-03-21T21:17:35.455 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:17:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[31986]: 2024-03-21T21:17:35.210+0000 7fb35dd82700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:17:35.456 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:35 smithi066 ceph-mon[25664]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:35.456 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:35 smithi066 ceph-mon[25664]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T21:17:35.456 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:35 smithi066 ceph-mon[25664]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T21:17:35.456 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:35 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:35.456 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:35 smithi066 ceph-mon[25664]: from='osd.5 [v2:172.21.15.66:6808/174977721,v1:172.21.15.66:6809/174977721]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:35.456 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:35 smithi066 ceph-mon[25664]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:35.456 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:35 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:35.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[24806]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[24806]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[24806]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[24806]: from='osd.5 [v2:172.21.15.66:6808/174977721,v1:172.21.15.66:6809/174977721]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[24806]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[29568]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[29568]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[29568]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T21:17:35.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:35.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[29568]: from='osd.5 [v2:172.21.15.66:6808/174977721,v1:172.21.15.66:6809/174977721]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:35.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[29568]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:35.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:35 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:36.014 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T21:17:36.014 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T21:17:36.014 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-03-21T21:17:36.014 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-03-21T21:17:36.014 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0169105 s, 620 MB/s 2024-03-21T21:17:36.015 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-03-21T21:17:36.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:36 smithi066 ceph-mon[25664]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:17:36.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:36 smithi066 ceph-mon[25664]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T21:17:36.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:36 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:36.412 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:36 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:36.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:36 smithi028 ceph-mon[29568]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:17:36.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:36 smithi028 ceph-mon[29568]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T21:17:36.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:36 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:36.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:36 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:36.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:36 smithi028 ceph-mon[24806]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:17:36.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:36 smithi028 ceph-mon[24806]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T21:17:36.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:36 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:36.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:36 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:37.338 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:37 smithi066 ceph-mon[25664]: purged_snaps scrub starts 2024-03-21T21:17:37.338 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:37 smithi066 ceph-mon[25664]: purged_snaps scrub ok 2024-03-21T21:17:37.338 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:37 smithi066 ceph-mon[25664]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:37.338 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:37 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:37.338 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:37 smithi066 ceph-mon[25664]: osd.5 [v2:172.21.15.66:6808/174977721,v1:172.21.15.66:6809/174977721] boot 2024-03-21T21:17:37.338 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:37 smithi066 ceph-mon[25664]: osdmap e38: 6 total, 6 up, 6 in 2024-03-21T21:17:37.338 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:37 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:37.339 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_2 2024-03-21T21:17:37.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[24806]: purged_snaps scrub starts 2024-03-21T21:17:37.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[24806]: purged_snaps scrub ok 2024-03-21T21:17:37.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[24806]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:37.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:37.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[24806]: osd.5 [v2:172.21.15.66:6808/174977721,v1:172.21.15.66:6809/174977721] boot 2024-03-21T21:17:37.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[24806]: osdmap e38: 6 total, 6 up, 6 in 2024-03-21T21:17:37.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:37.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[29568]: purged_snaps scrub starts 2024-03-21T21:17:37.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[29568]: purged_snaps scrub ok 2024-03-21T21:17:37.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[29568]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T21:17:37.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:37.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[29568]: osd.5 [v2:172.21.15.66:6808/174977721,v1:172.21.15.66:6809/174977721] boot 2024-03-21T21:17:37.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[29568]: osdmap e38: 6 total, 6 up, 6 in 2024-03-21T21:17:37.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:37 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:17:38.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:38 smithi066 ceph-mon[25664]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T21:17:38.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:38 smithi066 ceph-mon[25664]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T21:17:38.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:38 smithi028 ceph-mon[24806]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T21:17:38.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:38 smithi028 ceph-mon[24806]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T21:17:38.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:38 smithi028 ceph-mon[29568]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T21:17:38.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:38 smithi028 ceph-mon[29568]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T21:17:39.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:39.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T21:17:39.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:17:39.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:17:39.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:39.411 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:39.412 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:39.412 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:39 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:39.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:39.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:39.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T21:17:39.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:17:39.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:17:39.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:39.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:39.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:39.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:39.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:39 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:40.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:40 smithi066 ceph-mon[25664]: Detected new or changed devices on smithi066 2024-03-21T21:17:40.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:40 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi066 to 8544M 2024-03-21T21:17:40.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:40 smithi066 ceph-mon[25664]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T21:17:40.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:40 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:17:40.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:40 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:17:40.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:40 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:40.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[24806]: Detected new or changed devices on smithi066 2024-03-21T21:17:40.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi066 to 8544M 2024-03-21T21:17:40.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[24806]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T21:17:40.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:17:40.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:17:40.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:40.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[29568]: Detected new or changed devices on smithi066 2024-03-21T21:17:40.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi066 to 8544M 2024-03-21T21:17:40.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[29568]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T21:17:40.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:17:40.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:17:40.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:40 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:41.386 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:41 smithi066 ceph-mon[25664]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:17:41.387 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:41 smithi066 ceph-mon[25664]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:41.387 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:41 smithi066 ceph-mon[25664]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T21:17:41.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:41 smithi028 ceph-mon[29568]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:17:41.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:41 smithi028 ceph-mon[29568]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:41.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:41 smithi028 ceph-mon[29568]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T21:17:41.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:41 smithi028 ceph-mon[24806]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:17:41.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:41 smithi028 ceph-mon[24806]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:41.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:41 smithi028 ceph-mon[24806]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T21:17:43.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:43 smithi066 ceph-mon[25664]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:43.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:43 smithi066 ceph-mon[25664]: from='client.? 172.21.15.66:0/801448363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c33b4a2-8015-4917-a67b-be9ec57ed71b"}]: dispatch 2024-03-21T21:17:43.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:43 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c33b4a2-8015-4917-a67b-be9ec57ed71b"}]: dispatch 2024-03-21T21:17:43.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:43 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c33b4a2-8015-4917-a67b-be9ec57ed71b"}]': finished 2024-03-21T21:17:43.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:43 smithi066 ceph-mon[25664]: osdmap e43: 7 total, 6 up, 7 in 2024-03-21T21:17:43.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:43 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:43.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:43 smithi066 ceph-mon[25664]: from='client.? 172.21.15.66:0/2873047102' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:17:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[24806]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/801448363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c33b4a2-8015-4917-a67b-be9ec57ed71b"}]: dispatch 2024-03-21T21:17:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c33b4a2-8015-4917-a67b-be9ec57ed71b"}]: dispatch 2024-03-21T21:17:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c33b4a2-8015-4917-a67b-be9ec57ed71b"}]': finished 2024-03-21T21:17:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[24806]: osdmap e43: 7 total, 6 up, 7 in 2024-03-21T21:17:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/2873047102' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:17:43.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[29568]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:43.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/801448363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c33b4a2-8015-4917-a67b-be9ec57ed71b"}]: dispatch 2024-03-21T21:17:43.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c33b4a2-8015-4917-a67b-be9ec57ed71b"}]: dispatch 2024-03-21T21:17:43.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c33b4a2-8015-4917-a67b-be9ec57ed71b"}]': finished 2024-03-21T21:17:43.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[29568]: osdmap e43: 7 total, 6 up, 7 in 2024-03-21T21:17:43.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:43.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:43 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/2873047102' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:17:44.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:44 smithi028 ceph-mon[24806]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-21T21:17:44.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:44 smithi028 ceph-mon[24806]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-21T21:17:44.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:44 smithi028 ceph-mon[24806]: Cluster is now healthy 2024-03-21T21:17:44.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:44 smithi028 ceph-mon[29568]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-21T21:17:44.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:44 smithi028 ceph-mon[29568]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-21T21:17:44.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:44 smithi028 ceph-mon[29568]: Cluster is now healthy 2024-03-21T21:17:44.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:44 smithi066 ceph-mon[25664]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-21T21:17:44.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:44 smithi066 ceph-mon[25664]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-21T21:17:44.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:44 smithi066 ceph-mon[25664]: Cluster is now healthy 2024-03-21T21:17:47.381 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:47 smithi066 ceph-mon[25664]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-21T21:17:47.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:47 smithi028 ceph-mon[24806]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-21T21:17:47.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:47 smithi028 ceph-mon[29568]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-21T21:17:49.437 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:49 smithi066 ceph-mon[25664]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T21:17:49.437 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:49 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T21:17:49.437 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:49 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:49.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:49 smithi028 ceph-mon[24806]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T21:17:49.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:49 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T21:17:49.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:49 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:49.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:49 smithi028 ceph-mon[29568]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T21:17:49.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:49 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T21:17:49.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:49 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:50.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:50 smithi066 ceph-mon[25664]: Deploying daemon osd.6 on smithi066 2024-03-21T21:17:50.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:50 smithi028 ceph-mon[24806]: Deploying daemon osd.6 on smithi066 2024-03-21T21:17:50.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:50 smithi028 ceph-mon[29568]: Deploying daemon osd.6 on smithi066 2024-03-21T21:17:51.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:51 smithi066 ceph-mon[25664]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T21:17:51.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:51 smithi028 ceph-mon[24806]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T21:17:51.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:51 smithi028 ceph-mon[29568]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T21:17:53.295 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:53 smithi066 ceph-mon[25664]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T21:17:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:53 smithi028 ceph-mon[24806]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T21:17:53.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:53 smithi028 ceph-mon[29568]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T21:17:55.179 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:54 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:55.179 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:54 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:55.179 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:54 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:55.179 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:54 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:55.179 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:54 smithi066 ceph-mon[25664]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:55.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:55.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:55.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:55.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:55.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[29568]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:55.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:55.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:55.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:55.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:55.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:54 smithi028 ceph-mon[24806]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:57.262 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 6 on host 'smithi066' 2024-03-21T21:17:57.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:57 smithi066 ceph-mon[25664]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:57.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:57 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:57.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:57 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:57.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:57 smithi066 ceph-mon[25664]: from='osd.6 [v2:172.21.15.66:6816/1840826820,v1:172.21.15.66:6817/1840826820]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:17:57.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:57 smithi066 ceph-mon[25664]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:17:57.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[24806]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:57.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:57.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:57.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[24806]: from='osd.6 [v2:172.21.15.66:6816/1840826820,v1:172.21.15.66:6817/1840826820]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:17:57.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[24806]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:17:57.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[29568]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:57.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:57.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:57.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[29568]: from='osd.6 [v2:172.21.15.66:6816/1840826820,v1:172.21.15.66:6817/1840826820]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:17:57.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:57 smithi028 ceph-mon[29568]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:17:57.848 DEBUG:teuthology.orchestra.run.smithi066:osd.6> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.6.service 2024-03-21T21:17:57.851 INFO:tasks.cephadm:Deploying osd.7 on smithi066 with /dev/vg_nvme/lv_1... 2024-03-21T21:17:57.851 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T21:17:58.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:58.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:58.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:58.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:58.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-mon[25664]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T21:17:58.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-mon[25664]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T21:17:58.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:58.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-mon[25664]: from='osd.6 [v2:172.21.15.66:6816/1840826820,v1:172.21.15.66:6817/1840826820]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:58.449 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-mon[25664]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:58.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:58.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:58.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:58.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:58.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[24806]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T21:17:58.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[24806]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T21:17:58.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[24806]: from='osd.6 [v2:172.21.15.66:6816/1840826820,v1:172.21.15.66:6817/1840826820]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[24806]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[29568]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[29568]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[29568]: from='osd.6 [v2:172.21.15.66:6816/1840826820,v1:172.21.15.66:6817/1840826820]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:58.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:58 smithi028 ceph-mon[29568]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:17:58.742 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[34879]: 2024-03-21T21:17:58.454+0000 7f023b74e700 -1 osd.6 0 waiting for initial osdmap 2024-03-21T21:17:58.742 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:17:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[34879]: 2024-03-21T21:17:58.463+0000 7f02360e5700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:17:59.418 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:59 smithi066 ceph-mon[25664]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:59.418 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:59 smithi066 ceph-mon[25664]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:17:59.419 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:59 smithi066 ceph-mon[25664]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T21:17:59.419 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:59 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:59.419 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:17:59 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:59.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[24806]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:59.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[24806]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:17:59.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[24806]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T21:17:59.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:59.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:59.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[29568]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T21:17:59.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[29568]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:17:59.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[29568]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T21:17:59.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:17:59.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:17:59 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:18:00.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:00 smithi066 ceph-mon[25664]: purged_snaps scrub starts 2024-03-21T21:18:00.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:00 smithi066 ceph-mon[25664]: purged_snaps scrub ok 2024-03-21T21:18:00.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:00 smithi066 ceph-mon[25664]: osd.6 [v2:172.21.15.66:6816/1840826820,v1:172.21.15.66:6817/1840826820] boot 2024-03-21T21:18:00.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:00 smithi066 ceph-mon[25664]: osdmap e46: 7 total, 7 up, 7 in 2024-03-21T21:18:00.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:00 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:18:00.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:00 smithi066 ceph-mon[25664]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:00.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[29568]: purged_snaps scrub starts 2024-03-21T21:18:00.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[29568]: purged_snaps scrub ok 2024-03-21T21:18:00.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[29568]: osd.6 [v2:172.21.15.66:6816/1840826820,v1:172.21.15.66:6817/1840826820] boot 2024-03-21T21:18:00.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[29568]: osdmap e46: 7 total, 7 up, 7 in 2024-03-21T21:18:00.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:18:00.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[29568]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:00.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[24806]: purged_snaps scrub starts 2024-03-21T21:18:00.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[24806]: purged_snaps scrub ok 2024-03-21T21:18:00.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[24806]: osd.6 [v2:172.21.15.66:6816/1840826820,v1:172.21.15.66:6817/1840826820] boot 2024-03-21T21:18:00.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[24806]: osdmap e46: 7 total, 7 up, 7 in 2024-03-21T21:18:00.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:18:00.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:00 smithi028 ceph-mon[24806]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:01.247 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T21:18:01.248 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T21:18:01.248 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-03-21T21:18:01.248 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-03-21T21:18:01.248 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120443 s, 871 MB/s 2024-03-21T21:18:01.248 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-03-21T21:18:01.707 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:01 smithi066 ceph-mon[25664]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T21:18:01.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:01 smithi028 ceph-mon[24806]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T21:18:01.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:01 smithi028 ceph-mon[29568]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T21:18:02.318 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_1 2024-03-21T21:18:02.640 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T21:18:02.640 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: Detected new or changed devices on smithi066 2024-03-21T21:18:02.640 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:02.640 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:02.640 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:02.640 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:02.640 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi066 to 5696M 2024-03-21T21:18:02.640 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:02.640 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:02.641 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:02 smithi066 ceph-mon[25664]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:02.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: Detected new or changed devices on smithi066 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi066 to 5696M 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[24806]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:02.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T21:18:02.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: Detected new or changed devices on smithi066 2024-03-21T21:18:02.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:02.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:02.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:02.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:02.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi066 to 5696M 2024-03-21T21:18:02.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:02.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:02.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:02 smithi028 ceph-mon[29568]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:03.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:03 smithi066 ceph-mon[25664]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T21:18:03.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:03 smithi028 ceph-mon[24806]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T21:18:03.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:03 smithi028 ceph-mon[29568]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T21:18:04.677 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:04 smithi066 ceph-mon[25664]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T21:18:04.677 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:04 smithi066 ceph-mon[25664]: pgmap v155: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T21:18:04.677 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:04 smithi066 ceph-mon[25664]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:04.677 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:04 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:18:04.678 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:04 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:18:04.678 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:04 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:04.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[24806]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T21:18:04.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[24806]: pgmap v155: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T21:18:04.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[24806]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:04.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:18:04.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:18:04.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:04.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[29568]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T21:18:04.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[29568]: pgmap v155: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T21:18:04.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[29568]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:04.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T21:18:04.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T21:18:04.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:04 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:07.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:07 smithi066 ceph-mon[25664]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-21T21:18:07.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:07 smithi066 ceph-mon[25664]: from='client.? 172.21.15.66:0/934771445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c82af6d-0982-4624-a1ab-a60e1b2a7949"}]: dispatch 2024-03-21T21:18:07.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:07 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c82af6d-0982-4624-a1ab-a60e1b2a7949"}]: dispatch 2024-03-21T21:18:07.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:07 smithi066 ceph-mon[25664]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c82af6d-0982-4624-a1ab-a60e1b2a7949"}]': finished 2024-03-21T21:18:07.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:07 smithi066 ceph-mon[25664]: osdmap e51: 8 total, 7 up, 8 in 2024-03-21T21:18:07.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:07 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:07.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[24806]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-21T21:18:07.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/934771445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c82af6d-0982-4624-a1ab-a60e1b2a7949"}]: dispatch 2024-03-21T21:18:07.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c82af6d-0982-4624-a1ab-a60e1b2a7949"}]: dispatch 2024-03-21T21:18:07.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[24806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c82af6d-0982-4624-a1ab-a60e1b2a7949"}]': finished 2024-03-21T21:18:07.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[24806]: osdmap e51: 8 total, 7 up, 8 in 2024-03-21T21:18:07.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:07.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[29568]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-21T21:18:07.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/934771445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c82af6d-0982-4624-a1ab-a60e1b2a7949"}]: dispatch 2024-03-21T21:18:07.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c82af6d-0982-4624-a1ab-a60e1b2a7949"}]: dispatch 2024-03-21T21:18:07.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c82af6d-0982-4624-a1ab-a60e1b2a7949"}]': finished 2024-03-21T21:18:07.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[29568]: osdmap e51: 8 total, 7 up, 8 in 2024-03-21T21:18:07.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:07 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:08.491 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:08 smithi066 ceph-mon[25664]: from='client.? 172.21.15.66:0/1293494262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:18:08.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:08 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/1293494262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:18:08.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:08 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/1293494262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T21:18:09.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:09 smithi066 ceph-mon[25664]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T21:18:09.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:09 smithi028 ceph-mon[24806]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T21:18:09.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:09 smithi028 ceph-mon[29568]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T21:18:11.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:11 smithi066 ceph-mon[25664]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:11.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:11 smithi028 ceph-mon[24806]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:11.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:11 smithi028 ceph-mon[29568]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:13.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:13 smithi066 ceph-mon[25664]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:13.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:13 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T21:18:13.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:13 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:13 smithi028 ceph-mon[24806]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:13 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T21:18:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:13 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:13.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:13 smithi028 ceph-mon[29568]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:13.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:13 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T21:18:13.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:13 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:14.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:14 smithi066 ceph-mon[25664]: Deploying daemon osd.7 on smithi066 2024-03-21T21:18:14.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:14 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:18:14.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:14 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:18:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:14 smithi028 ceph-mon[24806]: Deploying daemon osd.7 on smithi066 2024-03-21T21:18:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:14 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:18:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:14 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:18:14.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:14 smithi028 ceph-mon[29568]: Deploying daemon osd.7 on smithi066 2024-03-21T21:18:14.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:14 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:18:14.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:14 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:18:15.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:15 smithi066 ceph-mon[25664]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:15 smithi028 ceph-mon[24806]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:15.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:15 smithi028 ceph-mon[29568]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:17.367 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:17 smithi066 ceph-mon[25664]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:17.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:17 smithi028 ceph-mon[24806]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:17.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:17 smithi028 ceph-mon[29568]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:19.552 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:19 smithi066 ceph-mon[25664]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:19.552 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:19 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:19.552 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:19 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:19.552 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:19 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:19.552 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:19 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:19.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[24806]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:19.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:19.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:19.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:19.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:19.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[29568]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:19.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:19.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:19.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:19.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:19 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:21.371 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 7 on host 'smithi066' 2024-03-21T21:18:21.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:21 smithi028 ceph-mon[24806]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:21 smithi028 ceph-mon[29568]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:21.690 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:21 smithi066 ceph-mon[25664]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:21.983 DEBUG:teuthology.orchestra.run.smithi066:osd.7> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.7.service 2024-03-21T21:18:21.985 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-21T21:18:21.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-21T21:18:22.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:22 smithi028 ceph-mon[24806]: from='osd.7 [v2:172.21.15.66:6824/2435919910,v1:172.21.15.66:6825/2435919910]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:18:22.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:22 smithi028 ceph-mon[24806]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:18:22.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:22 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:22.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:22 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:22.317 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:22 smithi028 ceph-mon[29568]: from='osd.7 [v2:172.21.15.66:6824/2435919910,v1:172.21.15.66:6825/2435919910]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:18:22.317 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:22 smithi028 ceph-mon[29568]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:18:22.317 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:22 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:22.317 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:22 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:22.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:22 smithi066 ceph-mon[25664]: from='osd.7 [v2:172.21.15.66:6824/2435919910,v1:172.21.15.66:6825/2435919910]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:18:22.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:22 smithi066 ceph-mon[25664]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:18:22.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:22 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:22.473 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:22 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:23.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:23 smithi066 ceph-mon[25664]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:23.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:23 smithi066 ceph-mon[25664]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T21:18:23.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:23 smithi066 ceph-mon[25664]: osdmap e52: 8 total, 7 up, 8 in 2024-03-21T21:18:23.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:23 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:23.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:23 smithi066 ceph-mon[25664]: from='osd.7 [v2:172.21.15.66:6824/2435919910,v1:172.21.15.66:6825/2435919910]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:18:23.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:23 smithi066 ceph-mon[25664]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:18:23.493 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:18:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[37801]: 2024-03-21T21:18:23.268+0000 7f0912470700 -1 osd.7 0 waiting for initial osdmap 2024-03-21T21:18:23.493 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:18:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[37801]: 2024-03-21T21:18:23.275+0000 7f090b604700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:18:23.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[29568]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:23.518 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[29568]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T21:18:23.518 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[29568]: osdmap e52: 8 total, 7 up, 8 in 2024-03-21T21:18:23.518 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:23.518 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[29568]: from='osd.7 [v2:172.21.15.66:6824/2435919910,v1:172.21.15.66:6825/2435919910]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:18:23.518 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[29568]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:18:23.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[24806]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:23.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[24806]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T21:18:23.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[24806]: osdmap e52: 8 total, 7 up, 8 in 2024-03-21T21:18:23.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:23.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[24806]: from='osd.7 [v2:172.21.15.66:6824/2435919910,v1:172.21.15.66:6825/2435919910]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:18:23.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:23 smithi028 ceph-mon[24806]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:18:24.196 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:18:24.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:24 smithi066 ceph-mon[25664]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:18:24.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:24 smithi066 ceph-mon[25664]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T21:18:24.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:24 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:24.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:24 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:24.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:24 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3639644362' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T21:18:24.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[24806]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:18:24.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[24806]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T21:18:24.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:24.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:24.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3639644362' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T21:18:24.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[29568]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-21T21:18:24.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[29568]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T21:18:24.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:24.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:24.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:24 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3639644362' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T21:18:25.122 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1711055879,"num_in_osds":8,"osd_in_since":1711055886,"num_remapped_pgs":0} 2024-03-21T21:18:25.378 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[29568]: purged_snaps scrub starts 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[29568]: purged_snaps scrub ok 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[29568]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[29568]: osd.7 [v2:172.21.15.66:6824/2435919910,v1:172.21.15.66:6825/2435919910] boot 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[29568]: osdmap e54: 8 total, 8 up, 8 in 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[24806]: purged_snaps scrub starts 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[24806]: purged_snaps scrub ok 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[24806]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[24806]: osd.7 [v2:172.21.15.66:6824/2435919910,v1:172.21.15.66:6825/2435919910] boot 2024-03-21T21:18:25.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[24806]: osdmap e54: 8 total, 8 up, 8 in 2024-03-21T21:18:25.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:25 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:25.471 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:25 smithi066 ceph-mon[25664]: purged_snaps scrub starts 2024-03-21T21:18:25.471 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:25 smithi066 ceph-mon[25664]: purged_snaps scrub ok 2024-03-21T21:18:25.471 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:25 smithi066 ceph-mon[25664]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T21:18:25.471 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:25 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:25.471 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:25 smithi066 ceph-mon[25664]: osd.7 [v2:172.21.15.66:6824/2435919910,v1:172.21.15.66:6825/2435919910] boot 2024-03-21T21:18:25.471 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:25 smithi066 ceph-mon[25664]: osdmap e54: 8 total, 8 up, 8 in 2024-03-21T21:18:25.471 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:25 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:26.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-21T21:18:26.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: osdmap e55: 8 total, 8 up, 8 in 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: osdmap e55: 8 total, 8 up, 8 in 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:26.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:26 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: osdmap e55: 8 total, 8 up, 8 in 2024-03-21T21:18:26.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:18:26.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:26.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:26.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:26.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:26.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:26 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:27.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:27 smithi028 ceph-mon[29568]: Detected new or changed devices on smithi066 2024-03-21T21:18:27.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:27 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi066 to 4272M 2024-03-21T21:18:27.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:27 smithi028 ceph-mon[29568]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:27.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:27 smithi028 ceph-mon[29568]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T21:18:27.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:27 smithi028 ceph-mon[24806]: Detected new or changed devices on smithi066 2024-03-21T21:18:27.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:27 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi066 to 4272M 2024-03-21T21:18:27.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:27 smithi028 ceph-mon[24806]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:27.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:27 smithi028 ceph-mon[24806]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T21:18:27.706 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:27 smithi066 ceph-mon[25664]: Detected new or changed devices on smithi066 2024-03-21T21:18:27.706 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:27 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi066 to 4272M 2024-03-21T21:18:27.706 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:27 smithi066 ceph-mon[25664]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:27.706 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:27 smithi066 ceph-mon[25664]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T21:18:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:18:28.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:28 smithi028 ceph-mon[29568]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T21:18:28.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:28 smithi028 ceph-mon[29568]: pgmap v174: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T21:18:28.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:28 smithi028 ceph-mon[24806]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T21:18:28.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:28 smithi028 ceph-mon[24806]: pgmap v174: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T21:18:28.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:28 smithi066 ceph-mon[25664]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T21:18:28.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:28 smithi066 ceph-mon[25664]: pgmap v174: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T21:18:29.093 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1711055904,"num_in_osds":8,"osd_in_since":1711055886,"num_remapped_pgs":0} 2024-03-21T21:18:29.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T21:18:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:18:29.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:29 smithi028 ceph-mon[29568]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T21:18:29.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:29 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/712005670' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T21:18:29.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:29 smithi028 ceph-mon[24806]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T21:18:29.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:29 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/712005670' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T21:18:29.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:29 smithi066 ceph-mon[25664]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T21:18:29.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:29 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/712005670' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T21:18:30.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:30 smithi028 ceph-mon[24806]: pgmap v176: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-21T21:18:30.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:30 smithi028 ceph-mon[29568]: pgmap v176: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-21T21:18:30.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:30 smithi066 ceph-mon[25664]: pgmap v176: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-21T21:18:31.226 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:18:31.226 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":58,"fsid":"c10cc158-e7c7-11ee-95cd-87774f69a715","created":"2024-03-21T21:13:16.822218+0000","modified":"2024-03-21T21:18:28.282410+0000","last_up_change":"2024-03-21T21:18:24.264759+0000","last_in_change":"2024-03-21T21:18:06.767778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T21:16:26.467699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff470de9-e261-403a-8a5e-be3156883903","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6803","nonce":3306214836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6805","nonce":3306214836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6809","nonce":3306214836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6807","nonce":3306214836}]},"public_addr":"172.21.15.28:6803/3306214836","cluster_addr":"172.21.15.28:6805/3306214836","heartbeat_back_addr":"172.21.15.28:6809/3306214836","heartbeat_front_addr":"172.21.15.28:6807/3306214836","state":["exists","up"]},{"osd":1,"uuid":"e259c2aa-88f8-4fcd-aba5-6323ad0e6e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6811","nonce":1005734520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6813","nonce":1005734520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6817","nonce":1005734520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6815","nonce":1005734520}]},"public_addr":"172.21.15.28:6811/1005734520","cluster_addr":"172.21.15.28:6813/1005734520","heartbeat_back_addr":"172.21.15.28:6817/1005734520","heartbeat_front_addr":"172.21.15.28:6815/1005734520","state":["exists","up"]},{"osd":2,"uuid":"8b6d94c3-b3f1-4274-a83c-7e8a1fa291aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6819","nonce":1632373792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6821","nonce":1632373792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6825","nonce":1632373792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6823","nonce":1632373792}]},"public_addr":"172.21.15.28:6819/1632373792","cluster_addr":"172.21.15.28:6821/1632373792","heartbeat_back_addr":"172.21.15.28:6825/1632373792","heartbeat_front_addr":"172.21.15.28:6823/1632373792","state":["exists","up"]},{"osd":3,"uuid":"c1bfc46b-7ec2-46e0-88e5-f417d2a01db7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6827","nonce":352039441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6829","nonce":352039441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6833","nonce":352039441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6831","nonce":352039441}]},"public_addr":"172.21.15.28:6827/352039441","cluster_addr":"172.21.15.28:6829/352039441","heartbeat_back_addr":"172.21.15.28:6833/352039441","heartbeat_front_addr":"172.21.15.28:6831/352039441","state":["exists","up"]},{"osd":4,"uuid":"d4efce96-eefe-4ed7-b490-dcef651634cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6801","nonce":1921306455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6803","nonce":1921306455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6807","nonce":1921306455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6805","nonce":1921306455}]},"public_addr":"172.21.15.66:6801/1921306455","cluster_addr":"172.21.15.66:6803/1921306455","heartbeat_back_addr":"172.21.15.66:6807/1921306455","heartbeat_front_addr":"172.21.15.66:6805/1921306455","state":["exists","up"]},{"osd":5,"uuid":"54c2da13-75c9-4c8a-93f6-d34110ac4cfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6809","nonce":174977721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6811","nonce":174977721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6815","nonce":174977721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6813","nonce":174977721}]},"public_addr":"172.21.15.66:6809/174977721","cluster_addr":"172.21.15.66:6811/174977721","heartbeat_back_addr":"172.21.15.66:6815/174977721","heartbeat_front_addr":"172.21.15.66:6813/174977721","state":["exists","up"]},{"osd":6,"uuid":"3c33b4a2-8015-4917-a67b-be9ec57ed71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6817","nonce":1840826820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6819","nonce":1840826820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6823","nonce":1840826820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6821","nonce":1840826820}]},"public_addr":"172.21.15.66:6817/1840826820","cluster_addr":"172.21.15.66:6819/1840826820","heartbeat_back_addr":"172.21.15.66:6823/1840826820","heartbeat_front_addr":"172.21.15.66:6821/1840826820","state":["exists","up"]},{"osd":7,"uuid":"0c82af6d-0982-4624-a1ab-a60e1b2a7949","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6825","nonce":2435919910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6827","nonce":2435919910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6831","nonce":2435919910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6829","nonce":2435919910}]},"public_addr":"172.21.15.66:6825/2435919910","cluster_addr":"172.21.15.66:6827/2435919910","heartbeat_back_addr":"172.21.15.66:6831/2435919910","heartbeat_front_addr":"172.21.15.66:6829/2435919910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:15:34.205328+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:15:59.927652+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:16:24.787987+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:16:49.530724+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:17:11.536360+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:17:34.418581+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:17:58.036484+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:18:22.353262+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3534476746":"2024-03-22T21:14:13.170135+0000","172.21.15.28:6801/2835376011":"2024-03-22T21:14:13.170135+0000","172.21.15.28:6800/2835376011":"2024-03-22T21:14:13.170135+0000","172.21.15.28:0/1248044138":"2024-03-22T21:14:13.170135+0000","172.21.15.28:0/3726932994":"2024-03-22T21:14:13.170135+0000","172.21.15.28:6801/2574647112":"2024-03-22T21:13:51.116367+0000","172.21.15.28:6800/2574647112":"2024-03-22T21:13:51.116367+0000","172.21.15.28:0/3956269744":"2024-03-22T21:13:51.116367+0000","172.21.15.28:0/413977725":"2024-03-22T21:13:51.116367+0000","172.21.15.28:0/1362282202":"2024-03-22T21:13:51.116367+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T21:18:31.529 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:31 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3249013487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T21:18:31.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:31 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3249013487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T21:18:31.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:31 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3249013487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T21:18:31.839 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T21:16:26.467699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-21T21:18:31.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-21T21:18:32.058 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:18:32.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:32 smithi028 ceph-mon[24806]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T21:18:32.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:32 smithi028 ceph-mon[29568]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T21:18:32.742 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:32 smithi066 ceph-mon[25664]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T21:18:34.021 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T21:18:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:34 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3580494205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T21:18:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:34 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3580494205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T21:18:34.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:34 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3580494205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T21:18:34.705 INFO:tasks.cephadm:Adding prometheus.a on smithi066 2024-03-21T21:18:34.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi066=a' 2024-03-21T21:18:35.288 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:35 smithi066 ceph-mon[25664]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T21:18:35.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:35 smithi028 ceph-mon[24806]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T21:18:35.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:35 smithi028 ceph-mon[29568]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T21:18:36.820 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled prometheus update... 2024-03-21T21:18:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[24806]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T21:18:37.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[29568]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T21:18:37.466 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-mon[25664]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:37.467 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:37.467 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:37.467 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:37.467 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:37.467 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' 2024-03-21T21:18:37.467 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T21:18:37.490 DEBUG:teuthology.orchestra.run.smithi066:prometheus.a> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-21T21:18:37.492 INFO:tasks.cephadm:Adding node-exporter.a on smithi028 2024-03-21T21:18:37.492 INFO:tasks.cephadm:Adding node-exporter.b on smithi066 2024-03-21T21:18:37.492 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi028=a;smithi066=b' 2024-03-21T21:18:38.170 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ignoring --setuser ceph since I am not root 2024-03-21T21:18:38.170 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ignoring --setgroup ceph since I am not root 2024-03-21T21:18:38.170 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:37 smithi066 ceph-mgr[26869]: -- 172.21.15.66:0/2908831603 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561664396340 con 0x56166436f000 2024-03-21T21:18:38.187 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ignoring --setuser ceph since I am not root 2024-03-21T21:18:38.187 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ignoring --setgroup ceph since I am not root 2024-03-21T21:18:38.187 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:37 smithi028 ceph-mgr[25042]: -- 172.21.15.28:0/3230557607 <== mon.1 v2:172.21.15.28:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56062963e340 con 0x560629616800 2024-03-21T21:18:38.438 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:38.398+0000 7f64c9e67000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:18:38.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-mon[29568]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi066=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:38.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-mon[29568]: Saving service prometheus spec with placement smithi066=a;count:1 2024-03-21T21:18:38.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-mon[29568]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T21:18:38.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-mon[29568]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T21:18:38.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-mon[24806]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi066=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:38.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-mon[24806]: Saving service prometheus spec with placement smithi066=a;count:1 2024-03-21T21:18:38.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-mon[24806]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T21:18:38.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-mon[24806]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T21:18:38.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:38 smithi066 ceph-mon[25664]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi066=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:38.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:38 smithi066 ceph-mon[25664]: Saving service prometheus spec with placement smithi066=a;count:1 2024-03-21T21:18:38.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:38 smithi066 ceph-mon[25664]: from='mgr.14152 172.21.15.28:0/3663517338' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T21:18:38.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:38 smithi066 ceph-mon[25664]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T21:18:38.493 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:38.424+0000 7ff2a5782000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:18:38.765 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:38.519+0000 7ff2a5782000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:18:38.938 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:38.489+0000 7f64c9e67000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:18:39.319 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:39.142+0000 7ff2a5782000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:18:39.319 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:39.236+0000 7ff2a5782000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:18:39.319 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:39.325+0000 7ff2a5782000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:18:39.417 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:39.095+0000 7f64c9e67000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:18:39.418 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:39.186+0000 7f64c9e67000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:18:39.418 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:39.272+0000 7f64c9e67000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:18:39.670 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:39.480+0000 7ff2a5782000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:18:39.670 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:39.564+0000 7ff2a5782000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:18:39.670 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:39.676+0000 7ff2a5782000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:18:39.688 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:39.423+0000 7f64c9e67000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:18:39.689 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:39.506+0000 7f64c9e67000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:18:39.689 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:39.617+0000 7f64c9e67000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:18:39.925 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:39.763+0000 7ff2a5782000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:18:39.950 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:39.703+0000 7f64c9e67000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:18:39.950 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:39.868+0000 7f64c9e67000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:18:40.242 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:39.931+0000 7ff2a5782000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:18:40.242 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:40 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:40.020+0000 7ff2a5782000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:18:40.438 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:39.957+0000 7f64c9e67000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:18:41.330 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:41.007+0000 7f64c9e67000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:18:41.331 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:41.141+0000 7f64c9e67000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:18:41.418 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:41.094+0000 7ff2a5782000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:18:41.418 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:41.229+0000 7ff2a5782000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:18:41.418 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:41.424+0000 7ff2a5782000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:18:41.688 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:41.338+0000 7f64c9e67000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:18:41.689 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:41.428+0000 7f64c9e67000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:18:41.689 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:41.568+0000 7f64c9e67000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:18:41.742 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:41.514+0000 7ff2a5782000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:18:41.742 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:41.654+0000 7ff2a5782000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:18:42.189 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:41.892+0000 7f64c9e67000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:18:42.189 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:42.120+0000 7f64c9e67000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:18:42.242 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:41.982+0000 7ff2a5782000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:18:42.242 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:42.210+0000 7ff2a5782000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:18:42.688 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:42.208+0000 7f64c9e67000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:18:42.689 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:42.383+0000 7f64c9e67000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:18:42.742 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:42.296+0000 7ff2a5782000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:18:42.742 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:42.472+0000 7ff2a5782000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:18:43.630 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:43.330+0000 7f64c9e67000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:18:43.630 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:43.637+0000 7f64c9e67000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:18:43.722 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:43.423+0000 7ff2a5782000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:18:43.886 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:43.736+0000 7f64c9e67000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:18:43.975 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:43.728+0000 7ff2a5782000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:18:43.975 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:43.826+0000 7ff2a5782000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:18:43.975 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:43.982+0000 7ff2a5782000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:18:44.182 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:43.893+0000 7f64c9e67000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:18:44.242 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: [21/Mar/2024:21:18:43] ENGINE Bus STARTING 2024-03-21T21:18:44.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: CherryPy Checker: 2024-03-21T21:18:44.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: The Application mounted at '' has an empty config. 2024-03-21T21:18:44.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: 2024-03-21T21:18:44.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: [21/Mar/2024:21:18:44] ENGINE Serving on http://:::9283 2024-03-21T21:18:44.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: [21/Mar/2024:21:18:44] ENGINE Bus STARTED 2024-03-21T21:18:44.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-mon[25664]: Active manager daemon y restarted 2024-03-21T21:18:44.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-mon[25664]: Activating manager daemon y 2024-03-21T21:18:44.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-mon[25664]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T21:18:44.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-mon[25664]: Standby manager daemon x restarted 2024-03-21T21:18:44.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-mon[25664]: Standby manager daemon x started 2024-03-21T21:18:44.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:18:44.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:18:44.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:18:44.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:44 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:18:44.439 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:18:44] ENGINE Bus STARTING 2024-03-21T21:18:44.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[24806]: Active manager daemon y restarted 2024-03-21T21:18:44.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[24806]: Activating manager daemon y 2024-03-21T21:18:44.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[24806]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[24806]: Standby manager daemon x restarted 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[24806]: Standby manager daemon x started 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[29568]: Active manager daemon y restarted 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[29568]: Activating manager daemon y 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[29568]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[29568]: Standby manager daemon x restarted 2024-03-21T21:18:44.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[29568]: Standby manager daemon x started 2024-03-21T21:18:44.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:18:44.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:18:44.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:18:44.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/1510209221' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:18:44.780 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: CherryPy Checker: 2024-03-21T21:18:44.780 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: The Application mounted at '' has an empty config. 2024-03-21T21:18:44.780 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:18:44.780 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:18:44] ENGINE Serving on http://:::9283 2024-03-21T21:18:44.780 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:18:44] ENGINE Bus STARTED 2024-03-21T21:18:44.780 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:18:44] ENGINE Bus STARTING 2024-03-21T21:18:45.188 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:18:44] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:18:45.188 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:18:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:18:44] ENGINE Bus STARTED 2024-03-21T21:18:45.210 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled node-exporter update... 2024-03-21T21:18:45.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: mgrmap e17: y(active, starting, since 0.2913s), standbys: x 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:18:45.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: Manager daemon y is now available 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:18:45.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:18:45.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: mgrmap e17: y(active, starting, since 0.2913s), standbys: x 2024-03-21T21:18:45.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:18:45.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:18:45.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:18:45.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:18:45.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:18:45.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:18:45.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:18:45.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: Manager daemon y is now available 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:45.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:45.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:45.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:18:45.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:18:45.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:18:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:18:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: [21/Mar/2024:21:18:44] ENGINE Bus STARTING 2024-03-21T21:18:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: [21/Mar/2024:21:18:44] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:18:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: [21/Mar/2024:21:18:44] ENGINE Bus STARTED 2024-03-21T21:18:45.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:18:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: [21/Mar/2024:21:18:44] ENGINE Bus STARTING 2024-03-21T21:18:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: [21/Mar/2024:21:18:44] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:18:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: [21/Mar/2024:21:18:44] ENGINE Bus STARTED 2024-03-21T21:18:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: mgrmap e17: y(active, starting, since 0.2913s), standbys: x 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:18:45.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: Manager daemon y is now available 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:18:45.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:45.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:18:45.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:18:45.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:18:45.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:18:45.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: [21/Mar/2024:21:18:44] ENGINE Bus STARTING 2024-03-21T21:18:45.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: [21/Mar/2024:21:18:44] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:18:45.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: [21/Mar/2024:21:18:44] ENGINE Bus STARTED 2024-03-21T21:18:45.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:45.880 DEBUG:teuthology.orchestra.run.smithi028:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-21T21:18:45.883 DEBUG:teuthology.orchestra.run.smithi066:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-21T21:18:45.885 INFO:tasks.cephadm:Adding alertmanager.a on smithi028 2024-03-21T21:18:45.885 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi028=a' 2024-03-21T21:18:46.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:46 smithi066 ceph-mon[25664]: mgrmap e18: y(active, since 1.29933s), standbys: x 2024-03-21T21:18:46.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:46 smithi066 ceph-mon[25664]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi066=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:46.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:46 smithi066 ceph-mon[25664]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:46.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:46 smithi066 ceph-mon[25664]: Saving service node-exporter spec with placement smithi028=a;smithi066=b;count:2 2024-03-21T21:18:46.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:46 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:46.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:46 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:46.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[24806]: mgrmap e18: y(active, since 1.29933s), standbys: x 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[24806]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi066=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[24806]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[24806]: Saving service node-exporter spec with placement smithi028=a;smithi066=b;count:2 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[29568]: mgrmap e18: y(active, since 1.29933s), standbys: x 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[29568]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi066=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[29568]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[29568]: Saving service node-exporter spec with placement smithi028=a;smithi066=b;count:2 2024-03-21T21:18:46.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:46.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:46 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:47.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:47 smithi066 ceph-mon[25664]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:47.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:47 smithi066 ceph-mon[25664]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-21T21:18:47.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:47 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:47.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:47 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:18:47.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:47 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:18:47.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:47 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[24806]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[24806]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[29568]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[29568]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:18:47.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:18:47.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:47 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:48.109 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled alertmanager update... 2024-03-21T21:18:48.661 DEBUG:teuthology.orchestra.run.smithi028:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-21T21:18:48.664 INFO:tasks.cephadm:Adding grafana.a on smithi066 2024-03-21T21:18:48.664 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi066=a' 2024-03-21T21:18:49.369 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:49 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:49.370 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:49 smithi066 ceph-mon[25664]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:49.370 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:49 smithi066 ceph-mon[25664]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-03-21T21:18:49.370 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:49 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:49.370 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:49 smithi066 ceph-mon[25664]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:49.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:49.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[24806]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:49.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[24806]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-03-21T21:18:49.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:49.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[24806]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:49.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:49.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[29568]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:49.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[29568]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-03-21T21:18:49.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:49.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:49 smithi028 ceph-mon[29568]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:50.423 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:50 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:50.424 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:50 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:18:50.424 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:50 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:18:50.424 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:50 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:50.424 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:50 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:50.424 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:50 smithi066 ceph-mon[25664]: Deploying daemon prometheus.a on smithi066 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[24806]: Deploying daemon prometheus.a on smithi066 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:18:50.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:50.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:50.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:50 smithi028 ceph-mon[29568]: Deploying daemon prometheus.a on smithi066 2024-03-21T21:18:50.817 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled grafana update... 2024-03-21T21:18:51.673 DEBUG:teuthology.orchestra.run.smithi066:grafana.a> sudo journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service 2024-03-21T21:18:51.676 INFO:tasks.cephadm:Setting up client nodes... 2024-03-21T21:18:51.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T21:18:51.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:51 smithi066 ceph-mon[25664]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:51.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:51 smithi066 ceph-mon[25664]: from='client.34373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi066=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:51.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:51 smithi066 ceph-mon[25664]: Saving service grafana spec with placement smithi066=a;count:1 2024-03-21T21:18:51.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:51 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:51.997 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:51 smithi028 ceph-mon[29568]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:51.997 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:51 smithi028 ceph-mon[29568]: from='client.34373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi066=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:51.998 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:51 smithi028 ceph-mon[29568]: Saving service grafana spec with placement smithi066=a;count:1 2024-03-21T21:18:51.998 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:51 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:51 smithi028 ceph-mon[24806]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:51 smithi028 ceph-mon[24806]: from='client.34373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi066=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:18:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:51 smithi028 ceph-mon[24806]: Saving service grafana spec with placement smithi066=a;count:1 2024-03-21T21:18:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:51 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:18:53.832 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T21:18:53.832 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQA9pPxl8ArWMRAAqR7fUPMt2v24kpGa9VUnog== 2024-03-21T21:18:54.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:53 smithi028 ceph-mon[24806]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:54.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:53 smithi028 ceph-mon[29568]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:54.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:53 smithi066 ceph-mon[25664]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:54.569 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T21:18:54.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-21T21:18:54.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-21T21:18:54.615 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T21:18:54.928 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:54 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/267120850' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T21:18:54.928 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:54 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/267120850' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T21:18:55.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:54 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/267120850' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T21:18:55.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:54 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/267120850' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T21:18:55.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:54 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/267120850' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T21:18:55.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:54 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/267120850' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T21:18:56.090 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:55 smithi028 ceph-mon[29568]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:56.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:55 smithi028 ceph-mon[24806]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:56.238 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:55 smithi066 ceph-mon[25664]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:57.669 INFO:teuthology.orchestra.run.smithi066.stdout:[client.1] 2024-03-21T21:18:57.669 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBBpPxlj08QKBAAuDhl+axekkiBP1osj5DJQA== 2024-03-21T21:18:57.992 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:57 smithi066 ceph-mon[25664]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:57.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:57 smithi066 ceph-mon[25664]: from='client.? 172.21.15.66:0/4181960561' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T21:18:57.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:57 smithi066 ceph-mon[25664]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T21:18:57.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:57 smithi066 ceph-mon[25664]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T21:18:58.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:57 smithi028 ceph-mon[24806]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:58.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:57 smithi028 ceph-mon[24806]: from='client.? 172.21.15.66:0/4181960561' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T21:18:58.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:57 smithi028 ceph-mon[24806]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T21:18:58.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:57 smithi028 ceph-mon[24806]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T21:18:58.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:57 smithi028 ceph-mon[29568]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:58.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:57 smithi028 ceph-mon[29568]: from='client.? 172.21.15.66:0/4181960561' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T21:18:58.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:57 smithi028 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T21:18:58.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:57 smithi028 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T21:18:59.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:18:59 smithi028 ceph-mon[29568]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:59.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:18:59 smithi028 ceph-mon[24806]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:18:59.492 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:18:59 smithi066 ceph-mon[25664]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:00.153 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-21T21:19:00.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-21T21:19:00.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-21T21:19:00.208 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-21T21:19:00.208 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-21T21:19:00.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-21T21:19:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:01.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-mon[25664]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:01.493 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 systemd[1]: Starting Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:19:01.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:01 smithi028 ceph-mon[29568]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:01.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:01 smithi028 ceph-mon[24806]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:01.758 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 podman[40799]: 2024-03-21 21:19:01.419737808 +0000 UTC m=+0.020279470 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T21:19:01.758 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 podman[40799]: 2024-03-21 21:19:01.547864322 +0000 UTC m=+0.148405981 container create 80c9222be6e9ce707a2f988d58b60d9f7616ea38470beebe5b1e2104dc36a2b5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 podman[40799]: 2024-03-21 21:19:01.733430404 +0000 UTC m=+0.333972066 container init 80c9222be6e9ce707a2f988d58b60d9f7616ea38470beebe5b1e2104dc36a2b5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 podman[40799]: 2024-03-21 21:19:01.735939614 +0000 UTC m=+0.336481274 container start 80c9222be6e9ce707a2f988d58b60d9f7616ea38470beebe5b1e2104dc36a2b5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.759Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.759Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.760Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.760Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi066 (none))" 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.760Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.760Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.761Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.761Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.763Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.765Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.765Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.701µs 2024-03-21T21:19:01.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.765Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T21:19:01.760 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.765Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-21T21:19:01.760 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.765Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.556µs wal_replay_duration=351.564µs total_replay_duration=391.444µs 2024-03-21T21:19:02.242 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.766Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T21:19:02.243 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.766Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T21:19:02.243 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.766Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T21:19:02.243 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.783Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.716391ms db_storage=695ns remote_storage=1.229µs web_handler=290ns query_engine=658ns scrape=413.128µs scrape_sd=18.257µs notify=812ns notify_sd=1.331µs rules=15.957675ms 2024-03-21T21:19:02.243 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:19:01.783Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T21:19:02.243 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 bash[40799]: 80c9222be6e9ce707a2f988d58b60d9f7616ea38470beebe5b1e2104dc36a2b5 2024-03-21T21:19:02.243 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:19:01 smithi066 systemd[1]: Started Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:19:02.439 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:19:02.689 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:02 smithi028 systemd[1]: Starting Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:19:03.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:02 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:03.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:02 smithi028 ceph-mon[24806]: Deploying daemon node-exporter.a on smithi028 2024-03-21T21:19:03.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:02 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/34297057' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T21:19:03.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:02 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:03.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:02 smithi028 ceph-mon[29568]: Deploying daemon node-exporter.a on smithi028 2024-03-21T21:19:03.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:02 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/34297057' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T21:19:03.193 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"active_gid":24373,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3101059707},{"type":"v1","addr":"172.21.15.28:6801","nonce":3101059707}]},"active_addr":"172.21.15.28:6801/3101059707","active_change":"2024-03-21T21:18:43.907297+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34328,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/","prometheus":"http://172.21.15.28:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":4172892718}]},{"addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1698649293}]},{"addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1106126851}]},{"addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1317986469}]}]}} 2024-03-21T21:19:03.196 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-21T21:19:03.196 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-21T21:19:03.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T21:19:03.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:02 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:03.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:02 smithi066 ceph-mon[25664]: Deploying daemon node-exporter.a on smithi028 2024-03-21T21:19:03.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:02 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/34297057' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T21:19:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:03.438 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:03 smithi028 bash[44478]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-21T21:19:04.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:03 smithi028 ceph-mon[24806]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:04.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:03 smithi028 ceph-mon[29568]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:04.189 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:03 smithi028 bash[44478]: Getting image source signatures 2024-03-21T21:19:04.189 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:03 smithi028 bash[44478]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-21T21:19:04.190 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:03 smithi028 bash[44478]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-21T21:19:04.190 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:03 smithi028 bash[44478]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-21T21:19:04.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:03 smithi066 ceph-mon[25664]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:05.589 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:05 smithi028 bash[44478]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-21T21:19:05.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:05 smithi028 ceph-mon[29568]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:05.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:05 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:05.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:05 smithi028 ceph-mon[24806]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:05.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:05 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:05.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:05 smithi066 ceph-mon[25664]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:05.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:05 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:05.910 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:05 smithi028 bash[44478]: Writing manifest to image destination 2024-03-21T21:19:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:19:06.199 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":59,"fsid":"c10cc158-e7c7-11ee-95cd-87774f69a715","created":"2024-03-21T21:13:16.822218+0000","modified":"2024-03-21T21:18:43.906983+0000","last_up_change":"2024-03-21T21:18:24.264759+0000","last_in_change":"2024-03-21T21:18:06.767778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T21:16:26.467699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff470de9-e261-403a-8a5e-be3156883903","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6803","nonce":3306214836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6805","nonce":3306214836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6809","nonce":3306214836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6807","nonce":3306214836}]},"public_addr":"172.21.15.28:6803/3306214836","cluster_addr":"172.21.15.28:6805/3306214836","heartbeat_back_addr":"172.21.15.28:6809/3306214836","heartbeat_front_addr":"172.21.15.28:6807/3306214836","state":["exists","up"]},{"osd":1,"uuid":"e259c2aa-88f8-4fcd-aba5-6323ad0e6e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6811","nonce":1005734520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6813","nonce":1005734520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6817","nonce":1005734520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6815","nonce":1005734520}]},"public_addr":"172.21.15.28:6811/1005734520","cluster_addr":"172.21.15.28:6813/1005734520","heartbeat_back_addr":"172.21.15.28:6817/1005734520","heartbeat_front_addr":"172.21.15.28:6815/1005734520","state":["exists","up"]},{"osd":2,"uuid":"8b6d94c3-b3f1-4274-a83c-7e8a1fa291aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6819","nonce":1632373792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6821","nonce":1632373792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6825","nonce":1632373792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6823","nonce":1632373792}]},"public_addr":"172.21.15.28:6819/1632373792","cluster_addr":"172.21.15.28:6821/1632373792","heartbeat_back_addr":"172.21.15.28:6825/1632373792","heartbeat_front_addr":"172.21.15.28:6823/1632373792","state":["exists","up"]},{"osd":3,"uuid":"c1bfc46b-7ec2-46e0-88e5-f417d2a01db7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6827","nonce":352039441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6829","nonce":352039441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6833","nonce":352039441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6831","nonce":352039441}]},"public_addr":"172.21.15.28:6827/352039441","cluster_addr":"172.21.15.28:6829/352039441","heartbeat_back_addr":"172.21.15.28:6833/352039441","heartbeat_front_addr":"172.21.15.28:6831/352039441","state":["exists","up"]},{"osd":4,"uuid":"d4efce96-eefe-4ed7-b490-dcef651634cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6801","nonce":1921306455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6803","nonce":1921306455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6807","nonce":1921306455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6805","nonce":1921306455}]},"public_addr":"172.21.15.66:6801/1921306455","cluster_addr":"172.21.15.66:6803/1921306455","heartbeat_back_addr":"172.21.15.66:6807/1921306455","heartbeat_front_addr":"172.21.15.66:6805/1921306455","state":["exists","up"]},{"osd":5,"uuid":"54c2da13-75c9-4c8a-93f6-d34110ac4cfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6809","nonce":174977721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6811","nonce":174977721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6815","nonce":174977721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6813","nonce":174977721}]},"public_addr":"172.21.15.66:6809/174977721","cluster_addr":"172.21.15.66:6811/174977721","heartbeat_back_addr":"172.21.15.66:6815/174977721","heartbeat_front_addr":"172.21.15.66:6813/174977721","state":["exists","up"]},{"osd":6,"uuid":"3c33b4a2-8015-4917-a67b-be9ec57ed71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6817","nonce":1840826820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6819","nonce":1840826820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6823","nonce":1840826820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6821","nonce":1840826820}]},"public_addr":"172.21.15.66:6817/1840826820","cluster_addr":"172.21.15.66:6819/1840826820","heartbeat_back_addr":"172.21.15.66:6823/1840826820","heartbeat_front_addr":"172.21.15.66:6821/1840826820","state":["exists","up"]},{"osd":7,"uuid":"0c82af6d-0982-4624-a1ab-a60e1b2a7949","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6825","nonce":2435919910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6827","nonce":2435919910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6831","nonce":2435919910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6829","nonce":2435919910}]},"public_addr":"172.21.15.66:6825/2435919910","cluster_addr":"172.21.15.66:6827/2435919910","heartbeat_back_addr":"172.21.15.66:6831/2435919910","heartbeat_front_addr":"172.21.15.66:6829/2435919910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:15:34.205328+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:15:59.927652+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:16:24.787987+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:16:49.530724+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:17:11.536360+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:17:34.418581+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:17:58.036484+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:18:22.353262+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/4016007355":"2024-03-22T21:18:43.906937+0000","172.21.15.28:0/1200163720":"2024-03-22T21:18:43.906937+0000","172.21.15.28:0/3354892894":"2024-03-22T21:18:43.906937+0000","172.21.15.28:0/3534476746":"2024-03-22T21:14:13.170135+0000","172.21.15.28:0/867242232":"2024-03-22T21:18:43.906937+0000","172.21.15.28:6801/2835376011":"2024-03-22T21:14:13.170135+0000","172.21.15.28:6800/2835376011":"2024-03-22T21:14:13.170135+0000","172.21.15.28:0/1248044138":"2024-03-22T21:14:13.170135+0000","172.21.15.28:0/3726932994":"2024-03-22T21:14:13.170135+0000","172.21.15.28:6801/2574647112":"2024-03-22T21:13:51.116367+0000","172.21.15.28:0/1888886278":"2024-03-22T21:18:43.906937+0000","172.21.15.28:6800/2574647112":"2024-03-22T21:13:51.116367+0000","172.21.15.28:0/3956269744":"2024-03-22T21:13:51.116367+0000","172.21.15.28:0/413977725":"2024-03-22T21:13:51.116367+0000","172.21.15.28:6800/4016007355":"2024-03-22T21:18:43.906937+0000","172.21.15.28:0/1362282202":"2024-03-22T21:13:51.116367+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T21:19:06.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1043905269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T21:19:06.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1043905269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T21:19:06.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 podman[44478]: 2024-03-21 21:19:06.197001706 +0000 UTC m=+2.923302067 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-21T21:19:06.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 podman[44478]: 2024-03-21 21:19:06.312714021 +0000 UTC m=+3.039014354 container create f01f375e6b149e9f9d18262547f2b748ff6c71db0ce33c391bff25e6e9fb23db (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:19:06.565 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:06 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1043905269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T21:19:06.937 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 podman[44478]: 2024-03-21 21:19:06.540284155 +0000 UTC m=+3.266584498 container init f01f375e6b149e9f9d18262547f2b748ff6c71db0ce33c391bff25e6e9fb23db (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:19:06.937 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 podman[44478]: 2024-03-21 21:19:06.542784802 +0000 UTC m=+3.269085132 container start f01f375e6b149e9f9d18262547f2b748ff6c71db0ce33c391bff25e6e9fb23db (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:19:06.937 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.546Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.546Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=arp 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-21T21:19:06.938 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-21T21:19:06.939 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=edac 2024-03-21T21:19:06.939 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-21T21:19:06.939 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-21T21:19:06.939 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-21T21:19:06.939 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-21T21:19:06.939 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-21T21:19:06.939 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-21T21:19:06.939 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=os 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-21T21:19:06.940 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=stat 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=time 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.547Z caller=node_exporter.go:115 level=info collector=uname 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.548Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.548Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.548Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.548Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[44650]: ts=2024-03-21T21:19:06.548Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 bash[44478]: f01f375e6b149e9f9d18262547f2b748ff6c71db0ce33c391bff25e6e9fb23db 2024-03-21T21:19:06.941 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:19:06 smithi028 systemd[1]: Started Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:19:07.349 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-21T21:19:07.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T21:19:07.402 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:07 smithi066 systemd[1]: Starting Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:19:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:07.677 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:07 smithi066 bash[41040]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-21T21:19:07.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:07 smithi028 ceph-mon[29568]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:07.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:07 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:07.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:07 smithi028 ceph-mon[29568]: Deploying daemon node-exporter.b on smithi066 2024-03-21T21:19:07.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:07 smithi028 ceph-mon[24806]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:07.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:07 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:07.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:07 smithi028 ceph-mon[24806]: Deploying daemon node-exporter.b on smithi066 2024-03-21T21:19:07.948 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:07 smithi066 ceph-mon[25664]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:07.948 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:07 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:07.948 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:07 smithi066 ceph-mon[25664]: Deploying daemon node-exporter.b on smithi066 2024-03-21T21:19:08.242 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:07 smithi066 bash[41040]: Getting image source signatures 2024-03-21T21:19:08.243 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:07 smithi066 bash[41040]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-21T21:19:08.243 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:07 smithi066 bash[41040]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-21T21:19:08.243 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:07 smithi066 bash[41040]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-21T21:19:09.457 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-mon[25664]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:09.457 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 bash[41040]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-21T21:19:09.592 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:19:09.593 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":59,"fsid":"c10cc158-e7c7-11ee-95cd-87774f69a715","created":"2024-03-21T21:13:16.822218+0000","modified":"2024-03-21T21:18:43.906983+0000","last_up_change":"2024-03-21T21:18:24.264759+0000","last_in_change":"2024-03-21T21:18:06.767778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T21:16:26.467699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff470de9-e261-403a-8a5e-be3156883903","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6803","nonce":3306214836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6805","nonce":3306214836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6809","nonce":3306214836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3306214836},{"type":"v1","addr":"172.21.15.28:6807","nonce":3306214836}]},"public_addr":"172.21.15.28:6803/3306214836","cluster_addr":"172.21.15.28:6805/3306214836","heartbeat_back_addr":"172.21.15.28:6809/3306214836","heartbeat_front_addr":"172.21.15.28:6807/3306214836","state":["exists","up"]},{"osd":1,"uuid":"e259c2aa-88f8-4fcd-aba5-6323ad0e6e24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6811","nonce":1005734520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6813","nonce":1005734520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6817","nonce":1005734520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1005734520},{"type":"v1","addr":"172.21.15.28:6815","nonce":1005734520}]},"public_addr":"172.21.15.28:6811/1005734520","cluster_addr":"172.21.15.28:6813/1005734520","heartbeat_back_addr":"172.21.15.28:6817/1005734520","heartbeat_front_addr":"172.21.15.28:6815/1005734520","state":["exists","up"]},{"osd":2,"uuid":"8b6d94c3-b3f1-4274-a83c-7e8a1fa291aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6819","nonce":1632373792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6821","nonce":1632373792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6825","nonce":1632373792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1632373792},{"type":"v1","addr":"172.21.15.28:6823","nonce":1632373792}]},"public_addr":"172.21.15.28:6819/1632373792","cluster_addr":"172.21.15.28:6821/1632373792","heartbeat_back_addr":"172.21.15.28:6825/1632373792","heartbeat_front_addr":"172.21.15.28:6823/1632373792","state":["exists","up"]},{"osd":3,"uuid":"c1bfc46b-7ec2-46e0-88e5-f417d2a01db7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6827","nonce":352039441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6829","nonce":352039441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6833","nonce":352039441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":352039441},{"type":"v1","addr":"172.21.15.28:6831","nonce":352039441}]},"public_addr":"172.21.15.28:6827/352039441","cluster_addr":"172.21.15.28:6829/352039441","heartbeat_back_addr":"172.21.15.28:6833/352039441","heartbeat_front_addr":"172.21.15.28:6831/352039441","state":["exists","up"]},{"osd":4,"uuid":"d4efce96-eefe-4ed7-b490-dcef651634cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6801","nonce":1921306455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6803","nonce":1921306455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6807","nonce":1921306455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":1921306455},{"type":"v1","addr":"172.21.15.66:6805","nonce":1921306455}]},"public_addr":"172.21.15.66:6801/1921306455","cluster_addr":"172.21.15.66:6803/1921306455","heartbeat_back_addr":"172.21.15.66:6807/1921306455","heartbeat_front_addr":"172.21.15.66:6805/1921306455","state":["exists","up"]},{"osd":5,"uuid":"54c2da13-75c9-4c8a-93f6-d34110ac4cfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6809","nonce":174977721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6811","nonce":174977721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6815","nonce":174977721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":174977721},{"type":"v1","addr":"172.21.15.66:6813","nonce":174977721}]},"public_addr":"172.21.15.66:6809/174977721","cluster_addr":"172.21.15.66:6811/174977721","heartbeat_back_addr":"172.21.15.66:6815/174977721","heartbeat_front_addr":"172.21.15.66:6813/174977721","state":["exists","up"]},{"osd":6,"uuid":"3c33b4a2-8015-4917-a67b-be9ec57ed71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6817","nonce":1840826820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6819","nonce":1840826820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6823","nonce":1840826820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":1840826820},{"type":"v1","addr":"172.21.15.66:6821","nonce":1840826820}]},"public_addr":"172.21.15.66:6817/1840826820","cluster_addr":"172.21.15.66:6819/1840826820","heartbeat_back_addr":"172.21.15.66:6823/1840826820","heartbeat_front_addr":"172.21.15.66:6821/1840826820","state":["exists","up"]},{"osd":7,"uuid":"0c82af6d-0982-4624-a1ab-a60e1b2a7949","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6825","nonce":2435919910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6827","nonce":2435919910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6830","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6831","nonce":2435919910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6828","nonce":2435919910},{"type":"v1","addr":"172.21.15.66:6829","nonce":2435919910}]},"public_addr":"172.21.15.66:6825/2435919910","cluster_addr":"172.21.15.66:6827/2435919910","heartbeat_back_addr":"172.21.15.66:6831/2435919910","heartbeat_front_addr":"172.21.15.66:6829/2435919910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:15:34.205328+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:15:59.927652+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:16:24.787987+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:16:49.530724+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:17:11.536360+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:17:34.418581+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:17:58.036484+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T21:18:22.353262+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/4016007355":"2024-03-22T21:18:43.906937+0000","172.21.15.28:0/1200163720":"2024-03-22T21:18:43.906937+0000","172.21.15.28:0/3354892894":"2024-03-22T21:18:43.906937+0000","172.21.15.28:0/3534476746":"2024-03-22T21:14:13.170135+0000","172.21.15.28:0/867242232":"2024-03-22T21:18:43.906937+0000","172.21.15.28:6801/2835376011":"2024-03-22T21:14:13.170135+0000","172.21.15.28:6800/2835376011":"2024-03-22T21:14:13.170135+0000","172.21.15.28:0/1248044138":"2024-03-22T21:14:13.170135+0000","172.21.15.28:0/3726932994":"2024-03-22T21:14:13.170135+0000","172.21.15.28:6801/2574647112":"2024-03-22T21:13:51.116367+0000","172.21.15.28:0/1888886278":"2024-03-22T21:18:43.906937+0000","172.21.15.28:6800/2574647112":"2024-03-22T21:13:51.116367+0000","172.21.15.28:0/3956269744":"2024-03-22T21:13:51.116367+0000","172.21.15.28:0/413977725":"2024-03-22T21:13:51.116367+0000","172.21.15.28:6800/4016007355":"2024-03-22T21:18:43.906937+0000","172.21.15.28:0/1362282202":"2024-03-22T21:13:51.116367+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T21:19:09.593 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:09 smithi028 ceph-mon[24806]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:09.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:09 smithi028 ceph-mon[29568]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:09.728 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 bash[41040]: Writing manifest to image destination 2024-03-21T21:19:09.979 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 podman[41040]: 2024-03-21 21:19:09.652370572 +0000 UTC m=+2.140109391 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-21T21:19:09.979 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 podman[41040]: 2024-03-21 21:19:09.792733034 +0000 UTC m=+2.280471839 container create c03593861fb95ff868b5231ecca09418e66333244131586826d3981760d50c69 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:19:10.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:10 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2199369831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T21:19:10.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:10 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:10.243 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 podman[41040]: 2024-03-21 21:19:09.986306969 +0000 UTC m=+2.474045776 container init c03593861fb95ff868b5231ecca09418e66333244131586826d3981760d50c69 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:19:10.243 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 podman[41040]: 2024-03-21 21:19:09.989414874 +0000 UTC m=+2.477153692 container start c03593861fb95ff868b5231ecca09418e66333244131586826d3981760d50c69 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:19:10.243 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.993Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-21T21:19:10.243 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.994Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.994Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.994Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=arp 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-21T21:19:10.244 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=edac 2024-03-21T21:19:10.245 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-21T21:19:10.245 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-21T21:19:10.245 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-21T21:19:10.245 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-21T21:19:10.245 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-21T21:19:10.245 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-21T21:19:10.245 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-21T21:19:10.245 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=os 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-21T21:19:10.246 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=stat 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=time 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=uname 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-21T21:19:10.247 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-21T21:19:10.248 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-21T21:19:10.248 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-21T21:19:10.248 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[41091]: ts=2024-03-21T21:19:09.995Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-21T21:19:10.248 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:10 smithi066 bash[41040]: c03593861fb95ff868b5231ecca09418e66333244131586826d3981760d50c69 2024-03-21T21:19:10.248 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:19:10 smithi066 systemd[1]: Started Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:19:10.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:10 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2199369831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T21:19:10.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:10 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:10.349 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:10 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2199369831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T21:19:10.349 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:10 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:10.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-21T21:19:10.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-21T21:19:10.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-21T21:19:10.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-21T21:19:10.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-21T21:19:10.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-21T21:19:10.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-21T21:19:10.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-21T21:19:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:11.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:11 smithi066 ceph-mon[25664]: Deploying daemon alertmanager.a on smithi028 2024-03-21T21:19:11.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:11 smithi066 ceph-mon[25664]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:11.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:11 smithi028 ceph-mon[29568]: Deploying daemon alertmanager.a on smithi028 2024-03-21T21:19:11.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:11 smithi028 ceph-mon[29568]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:11.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:11 smithi028 ceph-mon[24806]: Deploying daemon alertmanager.a on smithi028 2024-03-21T21:19:11.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:11 smithi028 ceph-mon[24806]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:11.960 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:11.961 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:11.961 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:12.789 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:19:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:12] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-03-21T21:19:13.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:19:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:19:13.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:13 smithi028 ceph-mon[29568]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:13.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:13 smithi028 ceph-mon[24806]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:13.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:13 smithi066 ceph-mon[25664]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:15.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:15 smithi028 ceph-mon[29568]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:15.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:15 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:15.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:15 smithi028 ceph-mon[24806]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:15.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:15 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:15.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:15 smithi066 ceph-mon[25664]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:15.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:15 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:16.729 INFO:teuthology.orchestra.run.smithi028.stdout:128849018906 2024-03-21T21:19:16.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-21T21:19:17.339 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:17 smithi028 ceph-mon[29568]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:17.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:17 smithi028 ceph-mon[24806]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:17.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:17 smithi066 ceph-mon[25664]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:19.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:19 smithi066 ceph-mon[25664]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:19.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:19 smithi028 ceph-mon[29568]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:19.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:19 smithi028 ceph-mon[24806]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:20.334 INFO:teuthology.orchestra.run.smithi028.stdout:55834574890 2024-03-21T21:19:20.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-21T21:19:21.573 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:21 smithi028 ceph-mon[29568]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:21.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:21 smithi028 ceph-mon[24806]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:21.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:21 smithi066 ceph-mon[25664]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:22.884 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:19:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:22] "GET /metrics HTTP/1.1" 200 191475 "" "Prometheus/2.33.4" 2024-03-21T21:19:23.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:19:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:19:23.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:23 smithi028 ceph-mon[29568]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:23.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:23 smithi028 ceph-mon[24806]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:23.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:23 smithi066 ceph-mon[25664]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:25.325 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:25.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:25 smithi028 ceph-mon[24806]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:25.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:25 smithi028 ceph-mon[29568]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:25.545 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:25 smithi066 ceph-mon[25664]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:26.386 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:27.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:27 smithi028 ceph-mon[24806]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:27.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:27 smithi028 ceph-mon[29568]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:27.434 INFO:teuthology.orchestra.run.smithi028.stdout:231928233998 2024-03-21T21:19:27.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-21T21:19:27.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:27 smithi066 ceph-mon[25664]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:28.471 INFO:teuthology.orchestra.run.smithi028.stdout:163208757273 2024-03-21T21:19:28.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-21T21:19:29.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:29 smithi028 ceph-mon[24806]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:29.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:29 smithi028 ceph-mon[29568]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:29.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:29 smithi066 ceph-mon[25664]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:29.504 INFO:teuthology.orchestra.run.smithi028.stdout:197568495635 2024-03-21T21:19:29.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-21T21:19:29.839 INFO:teuthology.orchestra.run.smithi028.stdout:107374182432 2024-03-21T21:19:29.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-21T21:19:31.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:31 smithi028 ceph-mon[24806]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:31.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:31 smithi028 ceph-mon[29568]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:31.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:31 smithi066 ceph-mon[25664]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:32.689 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:19:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:32] "GET /metrics HTTP/1.1" 200 191475 "" "Prometheus/2.33.4" 2024-03-21T21:19:32.776 INFO:teuthology.orchestra.run.smithi028.stdout:34359738416 2024-03-21T21:19:32.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-21T21:19:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:33.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:19:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:19:33.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:33 smithi028 ceph-mon[29568]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:33.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:33 smithi028 ceph-mon[24806]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:33.690 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:33 smithi028 systemd[1]: Starting Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:19:33.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:33 smithi066 ceph-mon[25664]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:35.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:35 smithi028 ceph-mon[29568]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:35.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:35 smithi028 ceph-mon[24806]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:35.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:35 smithi066 ceph-mon[25664]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:35.756 INFO:teuthology.orchestra.run.smithi028.stdout:128849018909 2024-03-21T21:19:36.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:36 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1364935332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T21:19:36.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:36 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1364935332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T21:19:36.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:36 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1364935332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T21:19:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:37 smithi028 ceph-mon[24806]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:37 smithi028 ceph-mon[29568]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:37.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:37 smithi066 ceph-mon[25664]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:39.057 INFO:teuthology.orchestra.run.smithi028.stdout:55834574894 2024-03-21T21:19:39.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:39 smithi028 ceph-mon[29568]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:39.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:39 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2639134648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T21:19:39.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:39 smithi028 ceph-mon[24806]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:39.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:39 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2639134648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T21:19:39.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:39 smithi066 ceph-mon[25664]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:39.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:39 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2639134648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T21:19:40.167 INFO:teuthology.orchestra.run.smithi028.stdout:77309411369 2024-03-21T21:19:40.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-21T21:19:40.582 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018906 got 128849018909 for osd.4 2024-03-21T21:19:40.582 DEBUG:teuthology.parallel:result is None 2024-03-21T21:19:41.404 INFO:teuthology.orchestra.run.smithi028.stdout:107374182436 2024-03-21T21:19:41.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:41 smithi028 ceph-mon[29568]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:41.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:41 smithi028 ceph-mon[24806]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:41.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:41 smithi066 ceph-mon[25664]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:42.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:42 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/292495954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T21:19:42.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:42 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/292495954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T21:19:42.690 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:19:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:42] "GET /metrics HTTP/1.1" 200 191475 "" "Prometheus/2.33.4" 2024-03-21T21:19:42.746 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182432 got 107374182436 for osd.3 2024-03-21T21:19:42.746 DEBUG:teuthology.parallel:result is None 2024-03-21T21:19:42.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:42 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/292495954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T21:19:42.991 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:42 smithi028 podman[46826]: 2024-03-21 21:19:42.764167122 +0000 UTC m=+4.383178794 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-21T21:19:42.992 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:42 smithi028 podman[46826]: 2024-03-21 21:19:42.898057203 +0000 UTC m=+4.517068848 volume create 51579b37b902d40f5ff65e56472893a1523eaf2e0f4225e6269ccb45b0eb7335 2024-03-21T21:19:42.992 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:42 smithi028 podman[46826]: 2024-03-21 21:19:42.998483598 +0000 UTC m=+4.617495243 container create 5d66b98a2084d96c994922eb10af4ecfbdde9b2a19af70f3276edd831b538f38 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:19:43.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:19:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:19:43.594 INFO:teuthology.orchestra.run.smithi028.stdout:34359738419 2024-03-21T21:19:43.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:43 smithi028 ceph-mon[24806]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:43.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:43 smithi028 ceph-mon[29568]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:43.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:43 smithi066 ceph-mon[25664]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:43.856 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574890 got 55834574894 for osd.1 2024-03-21T21:19:43.856 DEBUG:teuthology.parallel:result is None 2024-03-21T21:19:44.403 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/453212229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T21:19:44.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/453212229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T21:19:44.403 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 podman[46826]: 2024-03-21 21:19:44.131718557 +0000 UTC m=+5.750730206 container init 5d66b98a2084d96c994922eb10af4ecfbdde9b2a19af70f3276edd831b538f38 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:19:44.403 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 podman[46826]: 2024-03-21 21:19:44.134591255 +0000 UTC m=+5.753602904 container start 5d66b98a2084d96c994922eb10af4ecfbdde9b2a19af70f3276edd831b538f38 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:44.145Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:44.145Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:44.147Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.28 port=9094 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:44.148Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:44.173Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:44.174Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:44.176Z caller=main.go:518 msg=Listening address=:9093 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:44.176Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 bash[46826]: 5d66b98a2084d96c994922eb10af4ecfbdde9b2a19af70f3276edd831b538f38 2024-03-21T21:19:44.404 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:44 smithi028 systemd[1]: Started Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:19:44.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:44 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/453212229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T21:19:44.826 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738419 for osd.0 2024-03-21T21:19:44.826 DEBUG:teuthology.parallel:result is None 2024-03-21T21:19:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:45.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[24806]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:45.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[29568]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:19:45.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:19:45.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:19:45.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:19:45.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:45.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:19:45.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:19:45.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:19:45.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:45 smithi066 ceph-mon[25664]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:45.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:19:45.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:19:45.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:19:45.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:19:45.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:45.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:19:45.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:19:45.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:19:46.244 INFO:teuthology.orchestra.run.smithi028.stdout:197568495638 2024-03-21T21:19:46.412 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:46 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2258255354' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T21:19:46.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:46 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2258255354' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T21:19:46.413 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:46.148Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000446583s 2024-03-21T21:19:46.570 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:46 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2258255354' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T21:19:48.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:47 smithi028 ceph-mon[24806]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:48.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:47 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:48.049 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:47 smithi028 ceph-mon[29568]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:48.049 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:47 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:48.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:47 smithi066 ceph-mon[25664]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:48.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:47 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:48.652 INFO:teuthology.orchestra.run.smithi028.stdout:163208757277 2024-03-21T21:19:48.877 INFO:teuthology.orchestra.run.smithi028.stdout:231928234003 2024-03-21T21:19:49.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:49 smithi028 ceph-mon[24806]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:49.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:49 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/596559820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T21:19:49.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:49 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/724722362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T21:19:49.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:49 smithi028 ceph-mon[29568]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:49.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:49 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/596559820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T21:19:49.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:49 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/724722362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T21:19:49.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:49 smithi066 ceph-mon[25664]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:49.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:49 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/596559820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T21:19:49.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:49 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/724722362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T21:19:49.518 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757273 got 163208757277 for osd.5 2024-03-21T21:19:49.518 DEBUG:teuthology.parallel:result is None 2024-03-21T21:19:49.684 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495635 got 197568495638 for osd.6 2024-03-21T21:19:49.685 DEBUG:teuthology.parallel:result is None 2024-03-21T21:19:50.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:50 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:50.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:50 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:50.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:50 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:51.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233998 got 231928234003 for osd.7 2024-03-21T21:19:51.189 DEBUG:teuthology.parallel:result is None 2024-03-21T21:19:51.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:51 smithi028 ceph-mon[29568]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:51.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:51 smithi028 ceph-mon[24806]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:51.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:51 smithi066 ceph-mon[25664]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:51.887 INFO:teuthology.orchestra.run.smithi028.stdout:77309411372 2024-03-21T21:19:52.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:52 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/327328317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T21:19:52.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:52 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/327328317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T21:19:52.690 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:19:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:52] "GET /metrics HTTP/1.1" 200 191478 "" "Prometheus/2.33.4" 2024-03-21T21:19:52.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:52 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/327328317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T21:19:52.785 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411372 for osd.2 2024-03-21T21:19:52.785 DEBUG:teuthology.parallel:result is None 2024-03-21T21:19:52.786 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-21T21:19:52.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T21:19:53.243 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:19:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:19:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:19:53.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:53 smithi028 ceph-mon[24806]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:53.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:53 smithi028 ceph-mon[29568]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:53.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:53 smithi066 ceph-mon[25664]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:53.975 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:54.189 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:19:54.151Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002685927s 2024-03-21T21:19:55.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[24806]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:55.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[24806]: Deploying daemon grafana.a on smithi066 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[29568]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:19:55.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:19:55.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:54 smithi028 ceph-mon[29568]: Deploying daemon grafana.a on smithi066 2024-03-21T21:19:55.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:54 smithi066 ceph-mon[25664]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:55.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:19:55.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:54 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:19:55.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:19:55.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:54 smithi066 ceph-mon[25664]: Deploying daemon grafana.a on smithi066 2024-03-21T21:19:55.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:19:55.866 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T21:19:56.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:55 smithi066 ceph-mon[25664]: from='client.14637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T21:19:56.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:55 smithi028 ceph-mon[29568]: from='client.14637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T21:19:56.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:55 smithi028 ceph-mon[24806]: from='client.14637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T21:19:56.502 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-21T21:19:54.214427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48708,"kb_used_data":6084,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912508,"statfs":{"total":767960285184,"available":767910408192,"internally_reserved":0,"allocated":6230016,"data_stored":3488013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002502"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-21T21:18:44.299822+0000","last_change":"2024-03-21T21:18:28.290303+0000","last_active":"2024-03-21T21:18:44.299822+0000","last_peered":"2024-03-21T21:18:44.299822+0000","last_clean":"2024-03-21T21:18:44.299822+0000","last_became_active":"2024-03-21T21:18:28.289916+0000","last_became_peered":"2024-03-21T21:18:28.289916+0000","last_unstale":"2024-03-21T21:18:44.299822+0000","last_undegraded":"2024-03-21T21:18:44.299822+0000","last_fullsized":"2024-03-21T21:18:44.299822+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T21:16:27.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T21:16:27.197046+0000","last_clean_scrub_stamp":"2024-03-21T21:16:27.197046+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:48.377854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.441},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.626},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.376}]},{"osd":3,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.56499999999999995}]},{"osd":6,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.59699999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.432}]},{"osd":1,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.621}]},{"osd":2,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.39200000000000002}]},{"osd":3,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.497},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.66200000000000003}]},{"osd":4,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54100000000000004}]},{"osd":7,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.35799999999999998,"15min":0.33200000000000002},"min":{"1min":0.26800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.33300000000000002},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.47399999999999998}]},{"osd":2,"last update":"Thu Mar 21 21:19:32 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40300000000000002,"15min":0.38200000000000001},"min":{"1min":0.249,"5min":0.247,"15min":0.247},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.41999999999999998,"15min":0.40400000000000003},"min":{"1min":0.315,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.53800000000000003}]},{"osd":3,"last update":"Thu Mar 21 21:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.44,"15min":0.442},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.55600000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.496},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55100000000000005}]},{"osd":4,"last update":"Thu Mar 21 21:19:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.44400000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84099999999999997,"5min":1.002,"15min":1.002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.59299999999999997,"15min":0.58699999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.65500000000000003}]},{"osd":5,"last update":"Thu Mar 21 21:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.94199999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.78000000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.622}]},{"osd":6,"last update":"Thu Mar 21 21:19:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.70199999999999996}]},{"osd":7,"last update":"Thu Mar 21 21:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.72899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6856,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738296,"statfs":{"total":95995035648,"available":95988015104,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 21 21:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35099999999999998,"15min":0.33700000000000002},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.54200000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.307},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.35699999999999998,"15min":0.33900000000000002},"min":{"1min":0.27700000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.58299999999999996,"5min":0.626,"15min":0.626},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Mar 21 21:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.371,"15min":0.35699999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.58299999999999996,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.379},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56100000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.34300000000000003}]},{"osd":3,"last update":"Thu Mar 21 21:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.437,"15min":0.439},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55700000000000005,"5min":0.62,"15min":0.62},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45600000000000002,"15min":0.45700000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.57499999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.377}]},{"osd":4,"last update":"Thu Mar 21 21:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76100000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Mar 21 21:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.45000000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.66300000000000003}]},{"osd":6,"last update":"Thu Mar 21 21:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.54400000000000004}]},{"osd":7,"last update":"Thu Mar 21 21:19:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.626},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.55100000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6208,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738944,"statfs":{"total":95995035648,"available":95988678656,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.372,"15min":0.36699999999999999},"min":{"1min":0.251,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40100000000000002,"15min":0.39500000000000002},"min":{"1min":0.27400000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.434}]},{"osd":1,"last update":"Thu Mar 21 21:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39000000000000001,"15min":0.38400000000000001},"min":{"1min":0.28499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.377},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.42799999999999999,"15min":0.41699999999999998},"min":{"1min":0.28799999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Mar 21 21:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.42099999999999999,"15min":0.41799999999999998},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Mar 21 21:19:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57099999999999995,"15min":0.57699999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91700000000000004,"5min":1.012,"15min":1.012},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.61799999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93000000000000005,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Mar 21 21:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.623,"15min":0.627},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.84399999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.40500000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84499999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.72599999999999998}]},{"osd":6,"last update":"Thu Mar 21 21:19:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Mar 21 21:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63500000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29399999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40500000000000003,"15min":0.40300000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34899999999999998}]},{"osd":1,"last update":"Thu Mar 21 21:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40400000000000003,"15min":0.40500000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Mar 21 21:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45600000000000002,"15min":0.45700000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64600000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.38900000000000001}]},{"osd":4,"last update":"Thu Mar 21 21:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.55800000000000005,"15min":0.55000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.57599999999999996}]},{"osd":5,"last update":"Thu Mar 21 21:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.631,"5min":1.631,"15min":1.631},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.68899999999999995}]},{"osd":6,"last update":"Thu Mar 21 21:19:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":0.69799999999999995}]}]},{"osd":4,"up_from":30,"seq":128849018913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.46100000000000002}]},{"osd":1,"last update":"Thu Mar 21 21:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.44700000000000001}]},{"osd":2,"last update":"Thu Mar 21 21:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.39400000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Mar 21 21:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Mar 21 21:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.41899999999999998,"15min":0.40899999999999997},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.44700000000000001,"15min":0.42699999999999999},"min":{"1min":0.42499999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Mar 21 21:19:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Mar 21 21:19:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.70999999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5628,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739524,"statfs":{"total":95995035648,"available":95989272576,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65800000000000003,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.378,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78300000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.34699999999999998}]},{"osd":1,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72399999999999998,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.42299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84799999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67200000000000004,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.40300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.40500000000000003}]},{"osd":3,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81000000000000005,"5min":0.997,"15min":0.997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56000000000000005,"15min":0.55500000000000005},"min":{"1min":0.44500000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.41699999999999998,"15min":0.40500000000000003},"min":{"1min":0.312,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.44,"15min":0.42399999999999999},"min":{"1min":0.33800000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73499999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.63600000000000001}]},{"osd":6,"last update":"Thu Mar 21 21:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Mar 21 21:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.59999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T21:19:56.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T21:19:56.722 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:19:57.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:56 smithi028 ceph-mon[29568]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:57.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:56 smithi028 ceph-mon[24806]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:57.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:56 smithi066 ceph-mon[25664]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:58.571 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:19:58.572 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T21:19:59.272 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-21T21:19:58.215103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48708,"kb_used_data":6084,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912508,"statfs":{"total":767960285184,"available":767910408192,"internally_reserved":0,"allocated":6230016,"data_stored":3488013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002345"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-21T21:18:44.299822+0000","last_change":"2024-03-21T21:18:28.290303+0000","last_active":"2024-03-21T21:18:44.299822+0000","last_peered":"2024-03-21T21:18:44.299822+0000","last_clean":"2024-03-21T21:18:44.299822+0000","last_became_active":"2024-03-21T21:18:28.289916+0000","last_became_peered":"2024-03-21T21:18:28.289916+0000","last_unstale":"2024-03-21T21:18:44.299822+0000","last_undegraded":"2024-03-21T21:18:44.299822+0000","last_fullsized":"2024-03-21T21:18:44.299822+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T21:16:27.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T21:16:27.197046+0000","last_clean_scrub_stamp":"2024-03-21T21:16:27.197046+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:48.377854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.748},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.57499999999999996}]},{"osd":1,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.78500000000000003}]},{"osd":3,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.76900000000000002}]},{"osd":4,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.432},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.48599999999999999}]},{"osd":6,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.51400000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.623}]},{"osd":2,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.53900000000000003}]},{"osd":3,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.41799999999999998}]},{"osd":5,"last update":"Thu Mar 21 21:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.443},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.48799999999999999}]},{"osd":7,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.46999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.35799999999999998,"15min":0.33200000000000002},"min":{"1min":0.26800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.33300000000000002},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.308}]},{"osd":2,"last update":"Thu Mar 21 21:19:32 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40300000000000002,"15min":0.38200000000000001},"min":{"1min":0.249,"5min":0.247,"15min":0.247},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.375},{"interface":"front","average":{"1min":0.44,"5min":0.41999999999999998,"15min":0.40400000000000003},"min":{"1min":0.315,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Mar 21 21:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.44,"15min":0.442},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.55600000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Mar 21 21:19:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.44400000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84099999999999997,"5min":1.002,"15min":1.002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.59299999999999997,"15min":0.58699999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":5,"last update":"Thu Mar 21 21:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.94199999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.78000000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53000000000000003}]},{"osd":6,"last update":"Thu Mar 21 21:19:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.499}]},{"osd":7,"last update":"Thu Mar 21 21:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.57799999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6856,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738296,"statfs":{"total":95995035648,"available":95988015104,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 21 21:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35099999999999998,"15min":0.33700000000000002},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.54200000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.35699999999999998,"15min":0.33900000000000002},"min":{"1min":0.27700000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.58299999999999996,"5min":0.626,"15min":0.626},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Mar 21 21:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.371,"15min":0.35699999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.58299999999999996,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.379},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56100000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Mar 21 21:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.437,"15min":0.439},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55700000000000005,"5min":0.62,"15min":0.62},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45600000000000002,"15min":0.45700000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.57499999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Mar 21 21:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76100000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.69899999999999995}]},{"osd":5,"last update":"Thu Mar 21 21:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.45000000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.75900000000000001}]},{"osd":6,"last update":"Thu Mar 21 21:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Mar 21 21:19:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.68300000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6208,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738944,"statfs":{"total":95995035648,"available":95988678656,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.372,"15min":0.36699999999999999},"min":{"1min":0.251,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40100000000000002,"15min":0.39500000000000002},"min":{"1min":0.27400000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Mar 21 21:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39000000000000001,"15min":0.38400000000000001},"min":{"1min":0.28499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.436},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.42799999999999999,"15min":0.41699999999999998},"min":{"1min":0.28799999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Mar 21 21:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42099999999999999,"15min":0.41799999999999998},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Mar 21 21:19:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57099999999999995,"15min":0.57699999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91700000000000004,"5min":1.012,"15min":1.012},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.61799999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93000000000000005,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.502}]},{"osd":5,"last update":"Thu Mar 21 21:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.623,"15min":0.627},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.84399999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.40500000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84499999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Mar 21 21:19:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Mar 21 21:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.69799999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29399999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.375},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40500000000000003,"15min":0.40300000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34999999999999998}]},{"osd":1,"last update":"Thu Mar 21 21:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40400000000000003,"15min":0.40500000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Mar 21 21:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45600000000000002,"15min":0.45700000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64600000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.38900000000000001}]},{"osd":4,"last update":"Thu Mar 21 21:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.55800000000000005,"15min":0.55000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Mar 21 21:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.631,"5min":1.631,"15min":1.631},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Mar 21 21:19:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Mar 21 21:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":0.72699999999999998}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67100000000000004}]},{"osd":1,"last update":"Thu Mar 21 21:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.63500000000000001}]},{"osd":2,"last update":"Thu Mar 21 21:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.39400000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.69999999999999996}]},{"osd":3,"last update":"Thu Mar 21 21:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Mar 21 21:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.41899999999999998,"15min":0.40899999999999997},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.44700000000000001,"15min":0.42699999999999999},"min":{"1min":0.42499999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Mar 21 21:19:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.43099999999999999}]},{"osd":7,"last update":"Thu Mar 21 21:19:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52500000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5628,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739524,"statfs":{"total":95995035648,"available":95989272576,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65800000000000003,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.378,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78300000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72399999999999998,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.42299999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84799999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67200000000000004,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.40300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.66000000000000003}]},{"osd":3,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81000000000000005,"5min":0.997,"15min":0.997},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56000000000000005,"15min":0.55500000000000005},"min":{"1min":0.44500000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.754}]},{"osd":4,"last update":"Thu Mar 21 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.41699999999999998,"15min":0.40500000000000003},"min":{"1min":0.312,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.44,"15min":0.42399999999999999},"min":{"1min":0.33800000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73499999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.73799999999999999}]},{"osd":6,"last update":"Thu Mar 21 21:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Mar 21 21:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.78400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T21:19:59.274 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-21T21:19:59.274 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-21T21:19:59.274 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-21T21:19:59.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-21T21:19:59.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:59 smithi028 ceph-mon[29568]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:59.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:19:59 smithi028 ceph-mon[29568]: from='client.14643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T21:19:59.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:59 smithi028 ceph-mon[24806]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:59.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:19:59 smithi028 ceph-mon[24806]: from='client.14643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T21:19:59.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:59 smithi066 ceph-mon[25664]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:19:59.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:19:59 smithi066 ceph-mon[25664]: from='client.14643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T21:19:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T21:20:00.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:00 smithi066 ceph-mon[25664]: overall HEALTH_OK 2024-03-21T21:20:00.615 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:00 smithi028 ceph-mon[29568]: overall HEALTH_OK 2024-03-21T21:20:00.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:00 smithi028 ceph-mon[24806]: overall HEALTH_OK 2024-03-21T21:20:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:20:01.420 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-21T21:20:01.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:01 smithi066 ceph-mon[25664]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:01.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:01 smithi028 ceph-mon[29568]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:01.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:01 smithi028 ceph-mon[24806]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:02.128 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-21T21:20:02.128 INFO:tasks.cephadm:Setup complete, yielding 2024-03-21T21:20:02.129 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-21T21:20:02.140 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T21:20:02.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-03-21T21:20:02.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:02 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1236024104' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T21:20:02.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:02 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1236024104' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T21:20:02.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:02 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1236024104' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T21:20:02.794 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:20:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:02] "GET /metrics HTTP/1.1" 200 191478 "" "Prometheus/2.33.4" 2024-03-21T21:20:03.214 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:20:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:20:03.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:03 smithi028 ceph-mon[29568]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:03.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:03 smithi028 ceph-mon[24806]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:03.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:03 smithi066 ceph-mon[25664]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:05.375 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:20:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "id": "00a91bf2-4759-407f-9847-4572591b902a", 2024-03-21T21:20:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "r", 2024-03-21T21:20:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "current_period": "cd013c87-5f19-4a8a-bd4d-ad3f08eb29a0", 2024-03-21T21:20:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1 2024-03-21T21:20:05.376 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:20:05.598 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:05 smithi066 ceph-mon[25664]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:05.599 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:05 smithi066 ceph-mon[25664]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T21:20:05.599 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:05 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2101192053' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T21:20:05.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:05 smithi028 ceph-mon[29568]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:05.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:05 smithi028 ceph-mon[29568]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T21:20:05.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:05 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2101192053' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T21:20:05.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:05 smithi028 ceph-mon[24806]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:05.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:05 smithi028 ceph-mon[24806]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T21:20:05.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:05 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2101192053' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T21:20:06.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-03-21T21:20:06.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:06 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2101192053' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T21:20:06.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:06 smithi028 ceph-mon[29568]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T21:20:06.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:06 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2101192053' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T21:20:06.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:06 smithi028 ceph-mon[24806]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T21:20:06.709 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:06 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2101192053' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T21:20:06.710 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:06 smithi066 ceph-mon[25664]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T21:20:07.602 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:07 smithi066 ceph-mon[25664]: pgmap v46: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:07.602 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:07 smithi066 ceph-mon[25664]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T21:20:07.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:07 smithi028 ceph-mon[29568]: pgmap v46: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:07.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:07 smithi028 ceph-mon[29568]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T21:20:07.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:07 smithi028 ceph-mon[24806]: pgmap v46: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:07.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:07 smithi028 ceph-mon[24806]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T21:20:08.124 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:20:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "id": "11cce6a6-cc1a-4823-85f4-93eee7d798b8", 2024-03-21T21:20:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default", 2024-03-21T21:20:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "api_name": "default", 2024-03-21T21:20:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "is_master": "true", 2024-03-21T21:20:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "endpoints": [], 2024-03-21T21:20:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "hostnames": [], 2024-03-21T21:20:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "hostnames_s3website": [], 2024-03-21T21:20:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "master_zone": "", 2024-03-21T21:20:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "zones": [], 2024-03-21T21:20:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "placement_targets": [], 2024-03-21T21:20:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_placement": "", 2024-03-21T21:20:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "realm_id": "00a91bf2-4759-407f-9847-4572591b902a", 2024-03-21T21:20:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_policy": { 2024-03-21T21:20:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "groups": [] 2024-03-21T21:20:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:20:08.244 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 systemd[1]: Starting Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:20:08.574 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 podman[41783]: 2024-03-21 21:20:08.207928019 +0000 UTC m=+0.017440266 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T21:20:08.574 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 podman[41783]: 2024-03-21 21:20:08.319580128 +0000 UTC m=+0.129092431 container create a550093f473a589f57cd0a8b419e58abd666aa4950a680e7b10f571b12088c3c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, com.redhat.component=ubi8-container, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., maintainer=Paul Cuzner ) 2024-03-21T21:20:08.574 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 podman[41783]: 2024-03-21 21:20:08.546792103 +0000 UTC m=+0.356304360 container init a550093f473a589f57cd0a8b419e58abd666aa4950a680e7b10f571b12088c3c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , name=ubi8, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.buildah.version=1.24.2, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, vcs-type=git, vendor=Red Hat, Inc., version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-21T21:20:08.574 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 podman[41783]: 2024-03-21 21:20:08.549369763 +0000 UTC m=+0.358882018 container start a550093f473a589f57cd0a8b419e58abd666aa4950a680e7b10f571b12088c3c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, vendor=Red Hat, Inc., release=236.1648460182, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, architecture=x86_64, version=8.5, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, name=ubi8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container) 2024-03-21T21:20:08.574 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-21T21:20:08.574 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-21T21:20:08.823 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-21T21:20:08.823 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-21T21:20:08.823 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="App mode production" logger=settings 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-03-21T21:20:08.824 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-03-21T21:20:08.825 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-03-21T21:20:08.825 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-03-21T21:20:08.826 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-21T21:20:08.827 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-03-21T21:20:08.828 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-03-21T21:20:08.829 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 bash[41783]: a550093f473a589f57cd0a8b419e58abd666aa4950a680e7b10f571b12088c3c 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 systemd[1]: Started Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-03-21T21:20:08.830 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-21T21:20:08.831 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-03-21T21:20:08.832 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-03-21T21:20:08.833 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-03-21T21:20:08.834 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-03-21T21:20:08.835 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-03-21T21:20:08.836 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-03-21T21:20:08.837 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-03-21T21:20:08.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-03-21T21:20:09.074 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-03-21T21:20:09.075 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-03-21T21:20:09.076 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-03-21T21:20:09.077 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-03-21T21:20:09.077 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-03-21T21:20:09.078 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-03-21T21:20:09.079 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-03-21T21:20:09.080 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-03-21T21:20:09.081 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-03-21T21:20:09.082 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-03-21T21:20:09.083 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-21T21:20:09.084 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-21T21:20:09.085 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-03-21T21:20:09.086 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-03-21T21:20:09.086 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-03-21T21:20:09.086 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-03-21T21:20:09.086 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-03-21T21:20:09.086 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-21T21:20:09.086 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-03-21T21:20:09.463 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-03-21T21:20:09.463 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-21T21:20:09.463 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-03-21T21:20:09.463 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-21T21:20:09.463 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-21T21:20:09.463 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-03-21T21:20:09.464 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-03-21T21:20:09.465 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-03-21T21:20:09.466 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-03-21T21:20:09.466 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-03-21T21:20:09.466 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-03-21T21:20:09.466 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-03-21T21:20:09.466 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-03-21T21:20:09.466 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-03-21T21:20:09.466 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-03-21T21:20:09.466 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=557.02248ms 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-21T21:20:09.467 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-21T21:20:09.468 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-21T21:20:09.468 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-21T21:20:09.468 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-21T21:20:09.468 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-21T21:20:09.468 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-03-21T21:20:09.468 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-21T21:20:09.468 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-21T21:20:09.468 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-21T21:20:09.468 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:20:09+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-21T21:20:09.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[29568]: pgmap v48: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:09.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:09.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:09.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:09.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:09.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:09.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[24806]: pgmap v48: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:09.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:09.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:09.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:09.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:09.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:09 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:09.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-mon[25664]: pgmap v48: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:09.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:09.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:09.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:09.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:09.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:09 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:11.216 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:20:11.216 INFO:teuthology.orchestra.run.smithi028.stdout: "id": "f7770712-8db8-40ab-9bb1-9d69d790a1af", 2024-03-21T21:20:11.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-03-21T21:20:11.216 INFO:teuthology.orchestra.run.smithi028.stdout: "domain_root": "z.rgw.meta:root", 2024-03-21T21:20:11.216 INFO:teuthology.orchestra.run.smithi028.stdout: "control_pool": "z.rgw.control", 2024-03-21T21:20:11.216 INFO:teuthology.orchestra.run.smithi028.stdout: "gc_pool": "z.rgw.log:gc", 2024-03-21T21:20:11.216 INFO:teuthology.orchestra.run.smithi028.stdout: "lc_pool": "z.rgw.log:lc", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "log_pool": "z.rgw.log", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "roles_pool": "z.rgw.meta:roles", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "otp_pool": "z.rgw.otp", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "system_key": { 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "access_key": "", 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: "secret_key": "" 2024-03-21T21:20:11.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: "placement_pools": [ 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "default-placement", 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: "val": { 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: "index_pool": "z.rgw.buckets.index", 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_classes": { 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: "STANDARD": { 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: "data_pool": "z.rgw.buckets.data" 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: "index_type": 0 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:11.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:11.219 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:20:11.219 INFO:teuthology.orchestra.run.smithi028.stdout: "realm_id": "00a91bf2-4759-407f-9847-4572591b902a", 2024-03-21T21:20:11.219 INFO:teuthology.orchestra.run.smithi028.stdout: "notif_pool": "z.rgw.log:notif" 2024-03-21T21:20:11.219 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:20:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:11 smithi028 ceph-mon[29568]: pgmap v49: 33 pgs: 23 active+clean, 10 creating+peering; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1023 B/s wr, 1 op/s 2024-03-21T21:20:11.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:11 smithi028 ceph-mon[24806]: pgmap v49: 33 pgs: 23 active+clean, 10 creating+peering; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1023 B/s wr, 1 op/s 2024-03-21T21:20:11.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:11 smithi066 ceph-mon[25664]: pgmap v49: 33 pgs: 23 active+clean, 10 creating+peering; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1023 B/s wr, 1 op/s 2024-03-21T21:20:12.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-03-21T21:20:12.690 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:20:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:12] "GET /metrics HTTP/1.1" 200 191478 "" "Prometheus/2.33.4" 2024-03-21T21:20:13.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:13 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:13.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:13 smithi066 ceph-mon[25664]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-21T21:20:13.244 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:20:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:20:13.310 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:13 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:13.310 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:13 smithi028 ceph-mon[29568]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-21T21:20:13.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:13 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:13.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:13 smithi028 ceph-mon[24806]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-21T21:20:15.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[24806]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 461 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T21:20:15.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:15.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:15.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[24806]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T21:20:15.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[24806]: Reconfiguring daemon alertmanager.a on smithi028 2024-03-21T21:20:15.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[29568]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 461 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T21:20:15.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:15.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:15.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[29568]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T21:20:15.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:15 smithi028 ceph-mon[29568]: Reconfiguring daemon alertmanager.a on smithi028 2024-03-21T21:20:15.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:15 smithi066 ceph-mon[25664]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 461 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T21:20:15.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:15 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:15.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:15 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:15.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:15 smithi066 ceph-mon[25664]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T21:20:15.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:15 smithi066 ceph-mon[25664]: Reconfiguring daemon alertmanager.a on smithi028 2024-03-21T21:20:16.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:16 smithi028 ceph-mon[29568]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T21:20:16.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:16 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-21T21:20:16.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:16 smithi028 ceph-mon[24806]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T21:20:16.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:16 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-21T21:20:16.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:16 smithi066 ceph-mon[25664]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T21:20:16.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:16 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-21T21:20:17.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:17 smithi028 ceph-mon[24806]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-03-21T21:20:17.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:17 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-21T21:20:17.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:17 smithi028 ceph-mon[24806]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T21:20:17.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:17 smithi028 ceph-mon[29568]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-03-21T21:20:17.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:17 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-21T21:20:17.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:17 smithi028 ceph-mon[29568]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T21:20:17.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:17 smithi066 ceph-mon[25664]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-03-21T21:20:17.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:17 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-21T21:20:17.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:17 smithi066 ceph-mon[25664]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T21:20:18.316 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:18 smithi028 systemd[1]: Stopping Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:20:18.317 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:18 smithi028 bash[49454]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-21T21:20:18.667 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:18 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[47057]: level=info ts=2024-03-21T21:20:18.322Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:20:18.667 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:18 smithi028 podman[49467]: 2024-03-21 21:20:18.353099756 +0000 UTC m=+0.179378396 container died 5d66b98a2084d96c994922eb10af4ecfbdde9b2a19af70f3276edd831b538f38 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:18.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:18 smithi028 ceph-mon[24806]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T21:20:18.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:18 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-21T21:20:18.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:18 smithi028 ceph-mon[29568]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T21:20:18.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:18 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-21T21:20:18.941 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:18 smithi028 podman[49467]: 2024-03-21 21:20:18.77219846 +0000 UTC m=+0.598477110 container cleanup 5d66b98a2084d96c994922eb10af4ecfbdde9b2a19af70f3276edd831b538f38 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:18.941 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:18 smithi028 bash[49467]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T21:20:18.941 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:18 smithi028 bash[49493]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-21T21:20:18.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:18 smithi066 ceph-mon[25664]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T21:20:18.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:18 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-21T21:20:19.341 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 podman[49481]: 2024-03-21 21:20:19.015316633 +0000 UTC m=+0.658534215 container remove 5d66b98a2084d96c994922eb10af4ecfbdde9b2a19af70f3276edd831b538f38 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:19.341 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 podman[49481]: 2024-03-21 21:20:19.124336645 +0000 UTC m=+0.767554223 volume remove 51579b37b902d40f5ff65e56472893a1523eaf2e0f4225e6269ccb45b0eb7335 2024-03-21T21:20:19.341 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T21:20:19.341 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 systemd[1]: Stopped Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:20:19.341 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 systemd[1]: Starting Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:20:19.684 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 podman[49578]: 2024-03-21 21:20:19.24966509 +0000 UTC m=+0.018097333 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-21T21:20:19.684 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 podman[49578]: 2024-03-21 21:20:19.375154944 +0000 UTC m=+0.143587174 volume create e5cf90f4bd4d85f568e0426fd6abde58ea8d12e07df6ab528711dba0b7672859 2024-03-21T21:20:19.684 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 podman[49578]: 2024-03-21 21:20:19.484042062 +0000 UTC m=+0.252474334 container create 6d5934565000b0297e8a8e9203ef0bb72be71bf8a3bfae2d30801be2207322f3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:19.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:19 smithi066 ceph-mon[25664]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-21T21:20:19.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:19 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-21T21:20:19.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:19 smithi066 ceph-mon[25664]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T21:20:20.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-mon[29568]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-21T21:20:20.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-21T21:20:20.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-mon[29568]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T21:20:20.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-mon[24806]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-21T21:20:20.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2999236591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-21T21:20:20.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-mon[24806]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 podman[49578]: 2024-03-21 21:20:19.811056159 +0000 UTC m=+0.579488400 container init 6d5934565000b0297e8a8e9203ef0bb72be71bf8a3bfae2d30801be2207322f3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 podman[49578]: 2024-03-21 21:20:19.814019003 +0000 UTC m=+0.582451242 container start 6d5934565000b0297e8a8e9203ef0bb72be71bf8a3bfae2d30801be2207322f3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:19.824Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:19.825Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:19.826Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.28 port=9094 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:19.827Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:19.853Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:19.853Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:19.855Z caller=main.go:518 msg=Listening address=:9093 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:19.856Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 bash[49578]: 6d5934565000b0297e8a8e9203ef0bb72be71bf8a3bfae2d30801be2207322f3 2024-03-21T21:20:20.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:19 smithi028 systemd[1]: Started Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:20:20.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:20 smithi066 ceph-mon[25664]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T21:20:20.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:20 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T21:20:20.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:20 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:20.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:20 smithi066 ceph-mon[25664]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:20:20.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:20 smithi066 ceph-mon[25664]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:20:21.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[24806]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T21:20:21.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T21:20:21.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:21.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[24806]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:20:21.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[24806]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:20:21.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[29568]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T21:20:21.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T21:20:21.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:21.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[29568]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:20:21.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:20 smithi028 ceph-mon[29568]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:20:21.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:21 smithi066 ceph-mon[25664]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T21:20:21.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:21 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-21T21:20:21.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:21 smithi066 ceph-mon[25664]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T21:20:21.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:21 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T21:20:21.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:21 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T21:20:21.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:21 smithi066 ceph-mon[25664]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T21:20:21.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:21 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-21T21:20:22.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[29568]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T21:20:22.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-21T21:20:22.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[29568]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T21:20:22.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T21:20:22.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T21:20:22.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[29568]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T21:20:22.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-21T21:20:22.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:21.827Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000348366s 2024-03-21T21:20:22.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[24806]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T21:20:22.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-21T21:20:22.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[24806]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T21:20:22.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T21:20:22.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T21:20:22.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[24806]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T21:20:22.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:21 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-21T21:20:22.767 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:20:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:22] "GET /metrics HTTP/1.1" 200 192991 "" "Prometheus/2.33.4" 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout: "id": "21ec9aec-24d2-4aab-af45-af073f38cec0", 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout: "predecessor_uuid": "cd013c87-5f19-4a8a-bd4d-ad3f08eb29a0", 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_status": [], 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout: "period_map": { 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout: "id": "21ec9aec-24d2-4aab-af45-af073f38cec0", 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout: "zonegroups": [ 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T21:20:22.768 INFO:teuthology.orchestra.run.smithi028.stdout: "id": "11cce6a6-cc1a-4823-85f4-93eee7d798b8", 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default", 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "api_name": "default", 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "is_master": "true", 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "endpoints": [], 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "hostnames": [], 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "hostnames_s3website": [], 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "master_zone": "f7770712-8db8-40ab-9bb1-9d69d790a1af", 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "zones": [ 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "id": "f7770712-8db8-40ab-9bb1-9d69d790a1af", 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "endpoints": [], 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "log_meta": "false", 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "log_data": "false", 2024-03-21T21:20:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: "bucket_index_max_shards": 11, 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "read_only": "false", 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_type": "", 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_from_all": "true", 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_from": [], 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_zone": "" 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "placement_targets": [ 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default-placement", 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_classes": [ 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: "STANDARD" 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-21T21:20:22.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_placement": "default-placement", 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: "realm_id": "00a91bf2-4759-407f-9847-4572591b902a", 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_policy": { 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: "groups": [] 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: "short_zone_ids": [ 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "f7770712-8db8-40ab-9bb1-9d69d790a1af", 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 3994402005 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:20:22.771 INFO:teuthology.orchestra.run.smithi028.stdout: "master_zonegroup": "11cce6a6-cc1a-4823-85f4-93eee7d798b8", 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "master_zone": "f7770712-8db8-40ab-9bb1-9d69d790a1af", 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "period_config": { 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bucket_quota": { 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": false, 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "check_on_raw": false, 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max_size": -1, 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max_size_kb": 0, 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max_objects": -1 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "user_quota": { 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": false, 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "check_on_raw": false, 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max_size": -1, 2024-03-21T21:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max_size_kb": 0, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max_objects": -1 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "user_ratelimit": { 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max_read_ops": 0, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max_write_ops": 0, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max_read_bytes": 0, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max_write_bytes": 0, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": false 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "bucket_ratelimit": { 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max_read_ops": 0, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max_write_ops": 0, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max_read_bytes": 0, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max_write_bytes": 0, 2024-03-21T21:20:22.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": false 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: "anonymous_ratelimit": { 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max_read_ops": 0, 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max_write_ops": 0, 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max_read_bytes": 0, 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max_write_bytes": 0, 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": false 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: "realm_id": "00a91bf2-4759-407f-9847-4572591b902a", 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: "realm_name": "r", 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout: "realm_epoch": 2 2024-03-21T21:20:22.774 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:20:23.134 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:20:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:20:23.135 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:22 smithi066 systemd[1]: Stopping Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:20:23.135 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 bash[42477]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:20:23.410 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.140Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:20:23.410 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.140Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T21:20:23.410 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.140Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.140Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.140Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.140Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.140Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.140Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.140Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.141Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.141Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-21T21:20:23.141Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T21:20:23.411 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 podman[42490]: 2024-03-21 21:20:23.150698092 +0000 UTC m=+0.114598714 container died 80c9222be6e9ce707a2f988d58b60d9f7616ea38470beebe5b1e2104dc36a2b5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:23.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-03-21T21:20:23.705 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 podman[42490]: 2024-03-21 21:20:23.522962963 +0000 UTC m=+0.486863592 container cleanup 80c9222be6e9ce707a2f988d58b60d9f7616ea38470beebe5b1e2104dc36a2b5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:23.705 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 bash[42490]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T21:20:23.705 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 bash[42518]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:20:23.841 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:23 smithi028 ceph-mon[29568]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T21:20:23.841 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:23 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-21T21:20:23.841 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:23 smithi028 ceph-mon[29568]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T21:20:23.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:23 smithi028 ceph-mon[24806]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T21:20:23.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:23 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-21T21:20:23.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:23 smithi028 ceph-mon[24806]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T21:20:23.993 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-mon[25664]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T21:20:23.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2699473679' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-21T21:20:23.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:23 smithi066 ceph-mon[25664]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T21:20:23.994 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 podman[42505]: 2024-03-21 21:20:23.807892852 +0000 UTC m=+0.653890133 container remove 80c9222be6e9ce707a2f988d58b60d9f7616ea38470beebe5b1e2104dc36a2b5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:23.994 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T21:20:23.994 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 systemd[1]: Stopped Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:20:23.994 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:23 smithi066 systemd[1]: Starting Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:20:24.287 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 podman[42588]: 2024-03-21 21:20:24.042133244 +0000 UTC m=+0.109602926 container create c61488efcefc5e06d2403f1feec31090162c1f1b72456c7c79d549a5162ae62c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 podman[42588]: 2024-03-21 21:20:23.952464063 +0000 UTC m=+0.019933744 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 podman[42588]: 2024-03-21 21:20:24.260989896 +0000 UTC m=+0.328459587 container init c61488efcefc5e06d2403f1feec31090162c1f1b72456c7c79d549a5162ae62c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 podman[42588]: 2024-03-21 21:20:24.263773322 +0000 UTC m=+0.331243010 container start c61488efcefc5e06d2403f1feec31090162c1f1b72456c7c79d549a5162ae62c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.288Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.288Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.288Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.288Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi066 (none))" 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.288Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.288Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.290Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.290Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T21:20:24.288 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.291Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T21:20:24.540 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.293Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T21:20:24.540 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.294Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=25.409µs 2024-03-21T21:20:24.540 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.294Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T21:20:24.540 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 bash[42588]: c61488efcefc5e06d2403f1feec31090162c1f1b72456c7c79d549a5162ae62c 2024-03-21T21:20:24.540 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 systemd[1]: Started Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:20:24.540 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.517Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-21T21:20:24.540 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.518Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-21T21:20:24.540 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.518Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=58.178µs wal_replay_duration=224.073435ms total_replay_duration=224.429518ms 2024-03-21T21:20:24.540 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.522Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T21:20:24.541 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.522Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T21:20:24.541 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.522Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T21:20:24.541 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.545Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.633346ms db_storage=1.353µs remote_storage=2.691µs web_handler=562ns query_engine=1.095µs scrape=700.135µs scrape_sd=49.957µs notify=57.074µs notify_sd=31.72µs rules=22.327531ms 2024-03-21T21:20:24.541 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:20:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:20:24.545Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T21:20:25.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 226 B/s rd, 453 B/s wr, 1 op/s 2024-03-21T21:20:25.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:20:25.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.66:9095"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.66:9095"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.66:3000"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.66:3000"}]: dispatch 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:25.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:25.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:25 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:25.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 226 B/s rd, 453 B/s wr, 1 op/s 2024-03-21T21:20:25.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:20:25.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:20:25.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-03-21T21:20:25.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.66:9095"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.66:9095"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.66:3000"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.66:3000"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 226 B/s rd, 453 B/s wr, 1 op/s 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.66:9095"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.66:9095"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:20:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:20:25.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.66:3000"}]: dispatch 2024-03-21T21:20:25.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.66:3000"}]: dispatch 2024-03-21T21:20:25.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:25.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:25.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:25.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:25 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:25.737 INFO:teuthology.orchestra.run.smithi028.stdout:Scheduled rgw.foo update... 2024-03-21T21:20:26.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph osd pool create foo' 2024-03-21T21:20:26.763 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:26 smithi066 ceph-mon[25664]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:20:26.763 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:26 smithi066 ceph-mon[25664]: Saving service rgw.foo spec with placement count:2 2024-03-21T21:20:26.763 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:26 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:26.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:26 smithi028 ceph-mon[29568]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:20:26.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:26 smithi028 ceph-mon[29568]: Saving service rgw.foo spec with placement count:2 2024-03-21T21:20:26.803 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:26 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:26.803 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:26 smithi028 ceph-mon[24806]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:20:26.803 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:26 smithi028 ceph-mon[24806]: Saving service rgw.foo spec with placement count:2 2024-03-21T21:20:26.803 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:26 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:27.900 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:27 smithi066 ceph-mon[25664]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T21:20:28.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:27 smithi028 ceph-mon[29568]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T21:20:28.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:27 smithi028 ceph-mon[24806]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-21T21:20:29.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:28 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:29.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:28 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:29.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:28 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'foo' created 2024-03-21T21:20:30.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:29 smithi028 ceph-mon[29568]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 954 B/s wr, 2 op/s 2024-03-21T21:20:30.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:29 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/372371401' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-21T21:20:30.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:29 smithi028 ceph-mon[24806]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 954 B/s wr, 2 op/s 2024-03-21T21:20:30.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:29 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/372371401' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-21T21:20:30.191 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:20:29 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:20:29.830Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002569297s 2024-03-21T21:20:30.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:29 smithi066 ceph-mon[25664]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 954 B/s wr, 2 op/s 2024-03-21T21:20:30.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:29 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/372371401' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-21T21:20:30.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'rbd pool init foo' 2024-03-21T21:20:31.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:30 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/372371401' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-21T21:20:31.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:30 smithi028 ceph-mon[29568]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T21:20:31.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:30 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/372371401' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-21T21:20:31.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:30 smithi028 ceph-mon[24806]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T21:20:31.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:30 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/372371401' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-21T21:20:31.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:30 smithi066 ceph-mon[25664]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T21:20:32.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: pgmap v68: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-03-21T21:20:32.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T21:20:32.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:32.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:32.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T21:20:32.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:32.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: pgmap v68: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-03-21T21:20:32.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T21:20:32.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:32.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:32.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T21:20:32.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:31 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:32.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: pgmap v68: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-03-21T21:20:32.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T21:20:32.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.245 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:32.245 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:32.245 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T21:20:32.245 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:32.245 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:31 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:32.690 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:20:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:32] "GET /metrics HTTP/1.1" 200 196827 "" "Prometheus/2.33.4" 2024-03-21T21:20:33.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:32 smithi028 ceph-mon[24806]: Saving service rgw.foo spec with placement count:2 2024-03-21T21:20:33.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:32 smithi028 ceph-mon[24806]: Deploying daemon rgw.foo.smithi028.qubqiy on smithi028 2024-03-21T21:20:33.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:32 smithi028 ceph-mon[24806]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T21:20:33.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:32 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3710466560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-21T21:20:33.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:32 smithi028 ceph-mon[29568]: Saving service rgw.foo spec with placement count:2 2024-03-21T21:20:33.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:32 smithi028 ceph-mon[29568]: Deploying daemon rgw.foo.smithi028.qubqiy on smithi028 2024-03-21T21:20:33.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:32 smithi028 ceph-mon[29568]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T21:20:33.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:32 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3710466560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-21T21:20:33.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:32 smithi066 ceph-mon[25664]: Saving service rgw.foo spec with placement count:2 2024-03-21T21:20:33.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:32 smithi066 ceph-mon[25664]: Deploying daemon rgw.foo.smithi028.qubqiy on smithi028 2024-03-21T21:20:33.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:32 smithi066 ceph-mon[25664]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T21:20:33.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:32 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3710466560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-21T21:20:33.244 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:20:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:20:34.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:33 smithi028 ceph-mon[29568]: pgmap v71: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:34.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:33 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3710466560' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-21T21:20:34.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:33 smithi028 ceph-mon[29568]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T21:20:34.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:33 smithi028 ceph-mon[24806]: pgmap v71: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:34.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:33 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3710466560' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-21T21:20:34.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:33 smithi028 ceph-mon[24806]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T21:20:34.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:33 smithi066 ceph-mon[25664]: pgmap v71: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:34.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:33 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3710466560' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-21T21:20:34.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:33 smithi066 ceph-mon[25664]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T21:20:35.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:34 smithi028 ceph-mon[29568]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T21:20:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:34 smithi028 ceph-mon[24806]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T21:20:35.243 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:34 smithi066 ceph-mon[25664]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T21:20:35.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch apply iscsi foo u p' 2024-03-21T21:20:36.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:35 smithi066 ceph-mon[25664]: pgmap v74: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:36.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:35 smithi066 ceph-mon[25664]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T21:20:36.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:35 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:36.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:35 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:36.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:35 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:36.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:35 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T21:20:36.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:35 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:36.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:35 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:36.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[29568]: pgmap v74: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[29568]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[24806]: pgmap v74: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[24806]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:36.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:36.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:20:36.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T21:20:36.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:36.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:35 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:37.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:36 smithi028 ceph-mon[24806]: Deploying daemon rgw.foo.smithi066.gkaqro on smithi066 2024-03-21T21:20:37.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:36 smithi028 ceph-mon[29568]: Deploying daemon rgw.foo.smithi066.gkaqro on smithi066 2024-03-21T21:20:37.186 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:36 smithi066 ceph-mon[25664]: Deploying daemon rgw.foo.smithi066.gkaqro on smithi066 2024-03-21T21:20:37.960 INFO:teuthology.orchestra.run.smithi028.stdout:Scheduled iscsi.foo update... 2024-03-21T21:20:38.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:37 smithi028 ceph-mon[29568]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 8.2 KiB/s wr, 244 op/s 2024-03-21T21:20:38.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:37 smithi028 ceph-mon[24806]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 8.2 KiB/s wr, 244 op/s 2024-03-21T21:20:38.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:37 smithi066 ceph-mon[25664]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 8.2 KiB/s wr, 244 op/s 2024-03-21T21:20:38.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'sleep 180' 2024-03-21T21:20:39.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:38 smithi028 ceph-mon[29568]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:20:39.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:38 smithi028 ceph-mon[29568]: Saving service iscsi.foo spec with placement count:1 2024-03-21T21:20:39.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:38 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:39.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:38 smithi028 ceph-mon[29568]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T21:20:39.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:38 smithi028 ceph-mon[24806]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:20:39.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:38 smithi028 ceph-mon[24806]: Saving service iscsi.foo spec with placement count:1 2024-03-21T21:20:39.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:38 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:39.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:38 smithi028 ceph-mon[24806]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T21:20:39.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:38 smithi066 ceph-mon[25664]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:20:39.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:38 smithi066 ceph-mon[25664]: Saving service iscsi.foo spec with placement count:1 2024-03-21T21:20:39.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:38 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:39.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:38 smithi066 ceph-mon[25664]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T21:20:40.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:39 smithi066 ceph-mon[25664]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 6.6 KiB/s wr, 198 op/s 2024-03-21T21:20:40.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:39 smithi028 ceph-mon[29568]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 6.6 KiB/s wr, 198 op/s 2024-03-21T21:20:40.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:39 smithi028 ceph-mon[24806]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 6.6 KiB/s wr, 198 op/s 2024-03-21T21:20:41.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[29568]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.6 KiB/s wr, 178 op/s 2024-03-21T21:20:41.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:41.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:41.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:41.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:41.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[24806]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.6 KiB/s wr, 178 op/s 2024-03-21T21:20:41.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:41.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:41.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:41.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:41 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:42.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:41 smithi066 ceph-mon[25664]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.6 KiB/s wr, 178 op/s 2024-03-21T21:20:42.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:41 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:42.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:41 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:42.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:41 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:42.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:41 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:42.794 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:20:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:42] "GET /metrics HTTP/1.1" 200 196827 "" "Prometheus/2.33.4" 2024-03-21T21:20:43.244 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:20:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:20:44.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:43 smithi066 ceph-mon[25664]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-21T21:20:44.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:43 smithi028 ceph-mon[24806]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-21T21:20:44.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:43 smithi028 ceph-mon[29568]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-21T21:20:44.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-03-21T21:20:44.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-21T21:20:44.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-03-21T21:20:44.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-21T21:20:44.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-21T21:20:44.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-21T21:20:44.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-21T21:20:44.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-21T21:20:44.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:20:44.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:20:44.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:20:44.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:44.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:20:44.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:45.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:45.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:20:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:20:45.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:45.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:20:45.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:45.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:45 smithi066 ceph-mon[25664]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 3.8 KiB/s wr, 121 op/s 2024-03-21T21:20:45.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-03-21T21:20:45.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-03-21T21:20:45.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-21T21:20:45.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-21T21:20:45.994 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:45 smithi066 ceph-mon[25664]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T21:20:45.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:45.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:45 smithi066 ceph-mon[25664]: Checking dashboard <-> RGW credentials 2024-03-21T21:20:46.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[29568]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 3.8 KiB/s wr, 121 op/s 2024-03-21T21:20:46.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-03-21T21:20:46.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-03-21T21:20:46.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[29568]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[29568]: Checking dashboard <-> RGW credentials 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[24806]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 3.8 KiB/s wr, 121 op/s 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[24806]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:46.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:45 smithi028 ceph-mon[24806]: Checking dashboard <-> RGW credentials 2024-03-21T21:20:47.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 127 B/s wr, 134 op/s 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: Checking pool "foo" exists for service iscsi.foo 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[24806]: Deploying daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 127 B/s wr, 134 op/s 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: Checking pool "foo" exists for service iscsi.foo 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:47.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:46 smithi028 ceph-mon[29568]: Deploying daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:20:47.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T21:20:47.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 127 B/s wr, 134 op/s 2024-03-21T21:20:47.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:47.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:47.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:47.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: Checking pool "foo" exists for service iscsi.foo 2024-03-21T21:20:47.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:20:47.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:20:47.244 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T21:20:47.245 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:47.245 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:46 smithi066 ceph-mon[25664]: Deploying daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:20:49.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:49 smithi028 ceph-mon[24806]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 127 B/s wr, 124 op/s 2024-03-21T21:20:49.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:49 smithi028 ceph-mon[29568]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 127 B/s wr, 124 op/s 2024-03-21T21:20:49.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:49 smithi066 ceph-mon[25664]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 127 B/s wr, 124 op/s 2024-03-21T21:20:50.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:50 smithi066 ceph-mon[25664]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-21T21:20:50.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:50 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:50.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:50 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:50.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:50 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:50.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:50 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:50.563 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[29568]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-21T21:20:50.563 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:50.563 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:50.563 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:50.563 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:50.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[24806]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-21T21:20:50.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:50.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:50.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:50.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:50 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:51.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:51 smithi066 ceph-mon[25664]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 511 B/s wr, 131 op/s; 5 B/s, 0 objects/s recovering 2024-03-21T21:20:51.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:51 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2591774515' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:20:51.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:51 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/88278956' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4016007355"}]: dispatch 2024-03-21T21:20:51.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:51 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4016007355"}]: dispatch 2024-03-21T21:20:51.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:51 smithi028 ceph-mon[29568]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 511 B/s wr, 131 op/s; 5 B/s, 0 objects/s recovering 2024-03-21T21:20:51.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:51 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2591774515' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:20:51.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:51 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/88278956' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4016007355"}]: dispatch 2024-03-21T21:20:51.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:51 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4016007355"}]: dispatch 2024-03-21T21:20:51.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:51 smithi028 ceph-mon[24806]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 511 B/s wr, 131 op/s; 5 B/s, 0 objects/s recovering 2024-03-21T21:20:51.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:51 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2591774515' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:20:51.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:51 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/88278956' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4016007355"}]: dispatch 2024-03-21T21:20:51.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:51 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4016007355"}]: dispatch 2024-03-21T21:20:52.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:52 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4016007355"}]': finished 2024-03-21T21:20:52.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:52 smithi066 ceph-mon[25664]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T21:20:52.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:52 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3183253046' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1200163720"}]: dispatch 2024-03-21T21:20:52.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:52 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1200163720"}]: dispatch 2024-03-21T21:20:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:52 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4016007355"}]': finished 2024-03-21T21:20:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:52 smithi028 ceph-mon[29568]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T21:20:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:52 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3183253046' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1200163720"}]: dispatch 2024-03-21T21:20:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:52 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1200163720"}]: dispatch 2024-03-21T21:20:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:52 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4016007355"}]': finished 2024-03-21T21:20:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:52 smithi028 ceph-mon[24806]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T21:20:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:52 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3183253046' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1200163720"}]: dispatch 2024-03-21T21:20:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:52 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1200163720"}]: dispatch 2024-03-21T21:20:52.940 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:20:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:52] "GET /metrics HTTP/1.1" 200 207231 "" "Prometheus/2.33.4" 2024-03-21T21:20:53.227 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:20:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:20:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:20:53.404 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:53 smithi028 ceph-mon[24806]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 564 B/s wr, 145 op/s; 6 B/s, 0 objects/s recovering 2024-03-21T21:20:53.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:53 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1200163720"}]': finished 2024-03-21T21:20:53.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:53 smithi028 ceph-mon[24806]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T21:20:53.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:53 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/961607627' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3354892894"}]: dispatch 2024-03-21T21:20:53.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:53 smithi028 ceph-mon[29568]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 564 B/s wr, 145 op/s; 6 B/s, 0 objects/s recovering 2024-03-21T21:20:53.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:53 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1200163720"}]': finished 2024-03-21T21:20:53.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:53 smithi028 ceph-mon[29568]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T21:20:53.406 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:53 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/961607627' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3354892894"}]: dispatch 2024-03-21T21:20:53.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:53 smithi066 ceph-mon[25664]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 564 B/s wr, 145 op/s; 6 B/s, 0 objects/s recovering 2024-03-21T21:20:53.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:53 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1200163720"}]': finished 2024-03-21T21:20:53.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:53 smithi066 ceph-mon[25664]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T21:20:53.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:53 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/961607627' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3354892894"}]: dispatch 2024-03-21T21:20:54.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/961607627' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3354892894"}]': finished 2024-03-21T21:20:54.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T21:20:54.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/403286170' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3534476746"}]: dispatch 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: Adding iSCSI gateway http://:@172.21.15.28:5000 to Dashboard 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:54.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:54.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: Checking dashboard <-> RGW credentials 2024-03-21T21:20:54.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:54 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/961607627' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3354892894"}]': finished 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/403286170' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3534476746"}]: dispatch 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: Adding iSCSI gateway http://:@172.21.15.28:5000 to Dashboard 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:20:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:20:54.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:20:54.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:20:54.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:54.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:54.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:54.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: Checking dashboard <-> RGW credentials 2024-03-21T21:20:54.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/961607627' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3354892894"}]': finished 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/403286170' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3534476746"}]: dispatch 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: Adding iSCSI gateway http://:@172.21.15.28:5000 to Dashboard 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:20:54.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:20:54.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:20:54.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:54.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:20:54.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:20:54.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:20:54.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: Checking dashboard <-> RGW credentials 2024-03-21T21:20:54.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:54 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:55.654 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:55 smithi066 ceph-mon[25664]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 511 B/s wr, 9 op/s; 7 B/s, 0 objects/s recovering 2024-03-21T21:20:55.654 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:55 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:55.654 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:55 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/403286170' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3534476746"}]': finished 2024-03-21T21:20:55.654 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:55 smithi066 ceph-mon[25664]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T21:20:55.654 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:55 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2143236576' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/867242232"}]: dispatch 2024-03-21T21:20:55.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[24806]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 511 B/s wr, 9 op/s; 7 B/s, 0 objects/s recovering 2024-03-21T21:20:55.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:55.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/403286170' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3534476746"}]': finished 2024-03-21T21:20:55.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[24806]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T21:20:55.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2143236576' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/867242232"}]: dispatch 2024-03-21T21:20:55.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[29568]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 511 B/s wr, 9 op/s; 7 B/s, 0 objects/s recovering 2024-03-21T21:20:55.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:55.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/403286170' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3534476746"}]': finished 2024-03-21T21:20:55.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[29568]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T21:20:55.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:55 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2143236576' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/867242232"}]: dispatch 2024-03-21T21:20:56.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:56 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2143236576' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/867242232"}]': finished 2024-03-21T21:20:56.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:56 smithi028 ceph-mon[24806]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T21:20:56.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:56 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3878944418' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2835376011"}]: dispatch 2024-03-21T21:20:56.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:56 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2835376011"}]: dispatch 2024-03-21T21:20:56.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:56 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2143236576' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/867242232"}]': finished 2024-03-21T21:20:56.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:56 smithi028 ceph-mon[29568]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T21:20:56.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:56 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3878944418' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2835376011"}]: dispatch 2024-03-21T21:20:56.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:56 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2835376011"}]: dispatch 2024-03-21T21:20:56.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:56 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2143236576' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/867242232"}]': finished 2024-03-21T21:20:56.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:56 smithi066 ceph-mon[25664]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T21:20:56.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:56 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3878944418' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2835376011"}]: dispatch 2024-03-21T21:20:56.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:56 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2835376011"}]: dispatch 2024-03-21T21:20:57.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:57 smithi028 ceph-mon[29568]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-21T21:20:57.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:57 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2835376011"}]': finished 2024-03-21T21:20:57.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:57 smithi028 ceph-mon[29568]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T21:20:57.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:57 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2011006828' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2835376011"}]: dispatch 2024-03-21T21:20:57.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:57 smithi028 ceph-mon[24806]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-21T21:20:57.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:57 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2835376011"}]': finished 2024-03-21T21:20:57.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:57 smithi028 ceph-mon[24806]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T21:20:57.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:57 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2011006828' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2835376011"}]: dispatch 2024-03-21T21:20:57.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:57 smithi066 ceph-mon[25664]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-21T21:20:57.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:57 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2835376011"}]': finished 2024-03-21T21:20:57.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:57 smithi066 ceph-mon[25664]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T21:20:57.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:57 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2011006828' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2835376011"}]: dispatch 2024-03-21T21:20:58.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:58 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2011006828' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2835376011"}]': finished 2024-03-21T21:20:58.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:58 smithi028 ceph-mon[24806]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T21:20:58.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:58 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3839285968' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1248044138"}]: dispatch 2024-03-21T21:20:58.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:58 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1248044138"}]: dispatch 2024-03-21T21:20:58.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:58 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2011006828' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2835376011"}]': finished 2024-03-21T21:20:58.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:58 smithi028 ceph-mon[29568]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T21:20:58.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:58 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3839285968' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1248044138"}]: dispatch 2024-03-21T21:20:58.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:58 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1248044138"}]: dispatch 2024-03-21T21:20:58.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:58 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2011006828' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2835376011"}]': finished 2024-03-21T21:20:58.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:58 smithi066 ceph-mon[25664]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T21:20:58.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:58 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3839285968' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1248044138"}]: dispatch 2024-03-21T21:20:58.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:58 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1248044138"}]: dispatch 2024-03-21T21:20:59.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:59 smithi066 ceph-mon[25664]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-21T21:20:59.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:59 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1248044138"}]': finished 2024-03-21T21:20:59.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:59 smithi066 ceph-mon[25664]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T21:20:59.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:59 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/500881811' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3726932994"}]: dispatch 2024-03-21T21:20:59.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:20:59 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:59.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[29568]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-21T21:20:59.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1248044138"}]': finished 2024-03-21T21:20:59.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[29568]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T21:20:59.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/500881811' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3726932994"}]: dispatch 2024-03-21T21:20:59.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:20:59.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[24806]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-21T21:20:59.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1248044138"}]': finished 2024-03-21T21:20:59.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[24806]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T21:20:59.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/500881811' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3726932994"}]: dispatch 2024-03-21T21:20:59.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:20:59 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:21:00.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:00 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/500881811' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3726932994"}]': finished 2024-03-21T21:21:00.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:00 smithi066 ceph-mon[25664]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T21:21:00.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:00 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2465068740' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2574647112"}]: dispatch 2024-03-21T21:21:00.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:00 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/500881811' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3726932994"}]': finished 2024-03-21T21:21:00.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:00 smithi028 ceph-mon[24806]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T21:21:00.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:00 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2465068740' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2574647112"}]: dispatch 2024-03-21T21:21:00.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:00 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/500881811' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3726932994"}]': finished 2024-03-21T21:21:00.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:00 smithi028 ceph-mon[29568]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T21:21:00.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:00 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2465068740' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2574647112"}]: dispatch 2024-03-21T21:21:01.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:01 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:01.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:01 smithi066 ceph-mon[25664]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:01.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:01 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2465068740' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2574647112"}]': finished 2024-03-21T21:21:01.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:01 smithi066 ceph-mon[25664]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T21:21:01.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:01 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2052786421' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1888886278"}]: dispatch 2024-03-21T21:21:01.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:01.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[24806]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:01.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2465068740' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2574647112"}]': finished 2024-03-21T21:21:01.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[24806]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T21:21:01.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2052786421' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1888886278"}]: dispatch 2024-03-21T21:21:01.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:01.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[29568]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:01.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2465068740' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/2574647112"}]': finished 2024-03-21T21:21:01.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[29568]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T21:21:01.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:01 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2052786421' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1888886278"}]: dispatch 2024-03-21T21:21:02.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:02 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2052786421' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1888886278"}]': finished 2024-03-21T21:21:02.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:02 smithi066 ceph-mon[25664]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T21:21:02.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:02 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/538499651' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2574647112"}]: dispatch 2024-03-21T21:21:02.940 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:21:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:02] "GET /metrics HTTP/1.1" 200 208099 "" "Prometheus/2.33.4" 2024-03-21T21:21:02.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:02 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2052786421' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1888886278"}]': finished 2024-03-21T21:21:02.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:02 smithi028 ceph-mon[29568]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T21:21:02.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:02 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/538499651' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2574647112"}]: dispatch 2024-03-21T21:21:02.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:02 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2052786421' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1888886278"}]': finished 2024-03-21T21:21:02.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:02 smithi028 ceph-mon[24806]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T21:21:02.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:02 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/538499651' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2574647112"}]: dispatch 2024-03-21T21:21:03.244 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:21:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:21:03.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:03 smithi066 ceph-mon[25664]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:03.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:03 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/538499651' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2574647112"}]': finished 2024-03-21T21:21:03.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:03 smithi066 ceph-mon[25664]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T21:21:03.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:03 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3046737771' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3956269744"}]: dispatch 2024-03-21T21:21:03.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:03 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3956269744"}]: dispatch 2024-03-21T21:21:03.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[24806]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:03.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/538499651' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2574647112"}]': finished 2024-03-21T21:21:03.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[24806]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T21:21:03.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3046737771' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3956269744"}]: dispatch 2024-03-21T21:21:03.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3956269744"}]: dispatch 2024-03-21T21:21:03.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[29568]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:03.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/538499651' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/2574647112"}]': finished 2024-03-21T21:21:03.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[29568]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T21:21:03.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3046737771' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3956269744"}]: dispatch 2024-03-21T21:21:03.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:03 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3956269744"}]: dispatch 2024-03-21T21:21:04.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:04 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3956269744"}]': finished 2024-03-21T21:21:04.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:04 smithi066 ceph-mon[25664]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T21:21:04.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:04 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2954776009' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/413977725"}]: dispatch 2024-03-21T21:21:04.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:04 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/413977725"}]: dispatch 2024-03-21T21:21:04.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:04 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3956269744"}]': finished 2024-03-21T21:21:04.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:04 smithi028 ceph-mon[29568]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T21:21:04.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:04 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2954776009' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/413977725"}]: dispatch 2024-03-21T21:21:04.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:04 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/413977725"}]: dispatch 2024-03-21T21:21:04.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:04 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3956269744"}]': finished 2024-03-21T21:21:04.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:04 smithi028 ceph-mon[24806]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T21:21:04.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:04 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2954776009' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/413977725"}]: dispatch 2024-03-21T21:21:04.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:04 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/413977725"}]: dispatch 2024-03-21T21:21:05.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:05 smithi066 ceph-mon[25664]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:21:05.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:05 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/413977725"}]': finished 2024-03-21T21:21:05.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:05 smithi066 ceph-mon[25664]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T21:21:05.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:05 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/491499438' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4016007355"}]: dispatch 2024-03-21T21:21:05.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:05 smithi028 ceph-mon[24806]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:21:05.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:05 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/413977725"}]': finished 2024-03-21T21:21:05.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:05 smithi028 ceph-mon[24806]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T21:21:05.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:05 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/491499438' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4016007355"}]: dispatch 2024-03-21T21:21:05.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:05 smithi028 ceph-mon[29568]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:21:05.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:05 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/413977725"}]': finished 2024-03-21T21:21:05.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:05 smithi028 ceph-mon[29568]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T21:21:05.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:05 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/491499438' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4016007355"}]: dispatch 2024-03-21T21:21:06.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:06 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/491499438' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4016007355"}]': finished 2024-03-21T21:21:06.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:06 smithi066 ceph-mon[25664]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T21:21:06.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:06 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1038446559' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1362282202"}]: dispatch 2024-03-21T21:21:06.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:06 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/491499438' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4016007355"}]': finished 2024-03-21T21:21:06.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:06 smithi028 ceph-mon[29568]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T21:21:06.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:06 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1038446559' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1362282202"}]: dispatch 2024-03-21T21:21:06.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:06 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/491499438' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4016007355"}]': finished 2024-03-21T21:21:06.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:06 smithi028 ceph-mon[24806]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T21:21:06.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:06 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1038446559' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1362282202"}]: dispatch 2024-03-21T21:21:07.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:07 smithi066 ceph-mon[25664]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:07.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:07 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1038446559' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1362282202"}]': finished 2024-03-21T21:21:07.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:07 smithi066 ceph-mon[25664]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T21:21:07.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:07 smithi028 ceph-mon[24806]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:07.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:07 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1038446559' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1362282202"}]': finished 2024-03-21T21:21:07.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:07 smithi028 ceph-mon[24806]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T21:21:07.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:07 smithi028 ceph-mon[29568]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:07.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:07 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1038446559' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1362282202"}]': finished 2024-03-21T21:21:07.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:07 smithi028 ceph-mon[29568]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T21:21:09.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:09 smithi066 ceph-mon[25664]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:21:09.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:09 smithi028 ceph-mon[24806]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:21:09.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:09 smithi028 ceph-mon[29568]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:21:11.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:11 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:11 smithi028 ceph-mon[24806]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T21:21:11.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:11 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:11.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:11 smithi028 ceph-mon[29568]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T21:21:11.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:11 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:11.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:11 smithi066 ceph-mon[25664]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T21:21:12.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:21:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:12] "GET /metrics HTTP/1.1" 200 208099 "" "Prometheus/2.33.4" 2024-03-21T21:21:13.244 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:21:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:21:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:13 smithi028 ceph-mon[24806]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:21:13.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:13 smithi028 ceph-mon[29568]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:21:13.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:13 smithi066 ceph-mon[25664]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:21:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:15 smithi028 ceph-mon[24806]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-03-21T21:21:15.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:15 smithi028 ceph-mon[29568]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-03-21T21:21:15.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:15 smithi066 ceph-mon[25664]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-03-21T21:21:17.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:17 smithi028 ceph-mon[24806]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:21:17.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:17 smithi028 ceph-mon[29568]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:21:17.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:17 smithi066 ceph-mon[25664]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:21:19.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:19 smithi066 ceph-mon[25664]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-03-21T21:21:19.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:19 smithi028 ceph-mon[24806]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-03-21T21:21:19.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:19 smithi028 ceph-mon[29568]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-03-21T21:21:21.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:21 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:21.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:21 smithi028 ceph-mon[29568]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:21.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:21 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:21.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:21 smithi028 ceph-mon[24806]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:21.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:21 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:21.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:21 smithi066 ceph-mon[25664]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:22.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:21:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:22] "GET /metrics HTTP/1.1" 200 208088 "" "Prometheus/2.33.4" 2024-03-21T21:21:23.244 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:21:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:21:23.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:23 smithi028 ceph-mon[24806]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:23.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:23 smithi028 ceph-mon[29568]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:23.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:23 smithi066 ceph-mon[25664]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:25.547 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:25 smithi066 ceph-mon[25664]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:25.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:25 smithi028 ceph-mon[24806]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:25 smithi028 ceph-mon[29568]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:27.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:27 smithi028 ceph-mon[29568]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:27.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:27 smithi028 ceph-mon[24806]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:27.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:27 smithi066 ceph-mon[25664]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:29.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:29 smithi066 ceph-mon[25664]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:29.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:29 smithi028 ceph-mon[24806]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:29.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:29 smithi028 ceph-mon[29568]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:31.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:31 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:31.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:31 smithi028 ceph-mon[29568]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:31.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:31 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:31.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:31 smithi028 ceph-mon[24806]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:31.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:31 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:31.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:31 smithi066 ceph-mon[25664]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:32.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:21:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:32] "GET /metrics HTTP/1.1" 200 208028 "" "Prometheus/2.33.4" 2024-03-21T21:21:33.244 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:21:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:21:33.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:33 smithi028 ceph-mon[29568]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:33.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:33 smithi028 ceph-mon[24806]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:33.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:33 smithi066 ceph-mon[25664]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:35.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:35 smithi028 ceph-mon[29568]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:35.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:35 smithi028 ceph-mon[24806]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:35.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:35 smithi066 ceph-mon[25664]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:37.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:37 smithi028 ceph-mon[29568]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:37.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:37 smithi028 ceph-mon[24806]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:37.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:37 smithi066 ceph-mon[25664]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:39.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:39 smithi066 ceph-mon[25664]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:39.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:39 smithi028 ceph-mon[29568]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:39.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:39 smithi028 ceph-mon[24806]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:41.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:41 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:41.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:41 smithi028 ceph-mon[29568]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:41 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:41 smithi028 ceph-mon[24806]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:41.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:41 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:41.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:41 smithi066 ceph-mon[25664]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:42.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:21:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:42] "GET /metrics HTTP/1.1" 200 208028 "" "Prometheus/2.33.4" 2024-03-21T21:21:43.244 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:21:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:21:43.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:43 smithi028 ceph-mon[29568]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:43 smithi028 ceph-mon[24806]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:43.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:43 smithi066 ceph-mon[25664]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:44.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:21:44.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:21:44.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:21:44.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:21:44.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:21:44.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:21:45.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:45 smithi028 ceph-mon[24806]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:45.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:45 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:21:45.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:45 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:21:45.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:45 smithi028 ceph-mon[29568]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:45.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:45 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:21:45.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:45 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:21:45.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:45 smithi066 ceph-mon[25664]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:45.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:45 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:21:45.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:45 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:21:47.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:47 smithi028 ceph-mon[24806]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:47.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:47 smithi028 ceph-mon[29568]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:47.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:47 smithi066 ceph-mon[25664]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:49.494 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:49 smithi066 ceph-mon[25664]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:49.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:49 smithi028 ceph-mon[24806]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:49.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:49 smithi028 ceph-mon[29568]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:51.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:51 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:51.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:51 smithi028 ceph-mon[29568]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:51.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:51 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:51.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:51 smithi028 ceph-mon[24806]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:51.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:51 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:21:51.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:51 smithi066 ceph-mon[25664]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:52.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:21:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:52] "GET /metrics HTTP/1.1" 200 207980 "" "Prometheus/2.33.4" 2024-03-21T21:21:53.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:21:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:21:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:21:53.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:53 smithi028 ceph-mon[24806]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:53.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:53 smithi028 ceph-mon[29568]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:53.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:53 smithi066 ceph-mon[25664]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:54.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:21:54.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:21:54.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:54 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:21:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:21:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:21:54.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:54 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:21:54.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:21:54.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:21:54.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:54 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:21:55.646 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:55 smithi066 ceph-mon[25664]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:55.646 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:55 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:21:55.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:55 smithi028 ceph-mon[24806]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:55.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:55 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:21:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:55 smithi028 ceph-mon[29568]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:55 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:21:57.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:57 smithi028 ceph-mon[29568]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:57.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:57 smithi028 ceph-mon[24806]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:57.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:57 smithi066 ceph-mon[25664]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:21:59.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:21:59 smithi066 ceph-mon[25664]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:59.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:21:59 smithi028 ceph-mon[24806]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:21:59.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:21:59 smithi028 ceph-mon[29568]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:01.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:01 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:01.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:01 smithi028 ceph-mon[24806]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:01.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:01 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:01.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:01 smithi028 ceph-mon[29568]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:01.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:01 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:01.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:01 smithi066 ceph-mon[25664]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:02.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:22:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:02] "GET /metrics HTTP/1.1" 200 208005 "" "Prometheus/2.33.4" 2024-03-21T21:22:03.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:22:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:22:03.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:03 smithi028 ceph-mon[24806]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:03.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:03 smithi028 ceph-mon[29568]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:03.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:03 smithi066 ceph-mon[25664]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:05.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:05 smithi028 ceph-mon[24806]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:05.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:05 smithi028 ceph-mon[29568]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:05.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:05 smithi066 ceph-mon[25664]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:07.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:07 smithi028 ceph-mon[29568]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:07.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:07 smithi028 ceph-mon[24806]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:07.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:07 smithi066 ceph-mon[25664]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:09.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:09 smithi066 ceph-mon[25664]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:22:09.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:09 smithi028 ceph-mon[29568]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:22:09.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:09 smithi028 ceph-mon[24806]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:22:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:11 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:11 smithi028 ceph-mon[24806]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:11.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:11 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:11.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:11 smithi028 ceph-mon[29568]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:11.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:11 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:11.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:11 smithi066 ceph-mon[25664]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:12.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:22:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:12] "GET /metrics HTTP/1.1" 200 208005 "" "Prometheus/2.33.4" 2024-03-21T21:22:13.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:22:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:22:13.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:13 smithi028 ceph-mon[24806]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:22:13.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:13 smithi028 ceph-mon[29568]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:22:13.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:13 smithi066 ceph-mon[25664]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:22:15.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:15 smithi028 ceph-mon[29568]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:22:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:15 smithi028 ceph-mon[24806]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:22:15.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:15 smithi066 ceph-mon[25664]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:22:17.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:17 smithi028 ceph-mon[29568]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:17.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:17 smithi028 ceph-mon[24806]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:17.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:17 smithi066 ceph-mon[25664]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:19.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:19 smithi066 ceph-mon[25664]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:19.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:19 smithi028 ceph-mon[29568]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:19.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:19 smithi028 ceph-mon[24806]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:21.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:21 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:21.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:21 smithi028 ceph-mon[24806]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:21.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:21 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:21.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:21 smithi028 ceph-mon[29568]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:21.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:21 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:21.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:21 smithi066 ceph-mon[25664]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:22.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:22:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:22] "GET /metrics HTTP/1.1" 200 208002 "" "Prometheus/2.33.4" 2024-03-21T21:22:23.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:22:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:22:23.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:23 smithi028 ceph-mon[29568]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:23.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:23 smithi028 ceph-mon[24806]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:23.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:23 smithi066 ceph-mon[25664]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:25.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:25 smithi028 ceph-mon[29568]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:25 smithi028 ceph-mon[24806]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:25.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:25 smithi066 ceph-mon[25664]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:27.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:27 smithi028 ceph-mon[29568]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:27.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:27 smithi028 ceph-mon[24806]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:27.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:27 smithi066 ceph-mon[25664]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:29.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:29 smithi066 ceph-mon[25664]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:29.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:29 smithi028 ceph-mon[29568]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:29.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:29 smithi028 ceph-mon[24806]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:31.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:31 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:31.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:31 smithi028 ceph-mon[29568]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:31.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:31 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:31.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:31 smithi028 ceph-mon[24806]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:31.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:31 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:31.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:31 smithi066 ceph-mon[25664]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:32.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:22:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:32] "GET /metrics HTTP/1.1" 200 208006 "" "Prometheus/2.33.4" 2024-03-21T21:22:33.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:22:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:22:33.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:33 smithi028 ceph-mon[24806]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:33.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:33 smithi028 ceph-mon[29568]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:33.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:33 smithi066 ceph-mon[25664]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:35.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:35 smithi028 ceph-mon[24806]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:35.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:35 smithi028 ceph-mon[29568]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:35.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:35 smithi066 ceph-mon[25664]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:37.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:37 smithi028 ceph-mon[24806]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:37.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:37 smithi028 ceph-mon[29568]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:37.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:37 smithi066 ceph-mon[25664]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:39.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:39 smithi066 ceph-mon[25664]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:39.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:39 smithi028 ceph-mon[24806]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:39.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:39 smithi028 ceph-mon[29568]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:41 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:41.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:41 smithi028 ceph-mon[24806]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:41 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:41 smithi028 ceph-mon[29568]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:41.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:41 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:41.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:41 smithi066 ceph-mon[25664]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:42.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:22:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:42] "GET /metrics HTTP/1.1" 200 208006 "" "Prometheus/2.33.4" 2024-03-21T21:22:43.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:22:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:22:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:43 smithi028 ceph-mon[24806]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:43 smithi028 ceph-mon[29568]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:43.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:43 smithi066 ceph-mon[25664]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:44.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:22:44.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:22:44.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:22:44.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:22:44.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:22:44.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:22:44.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:22:44.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:22:44.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:22:44.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:22:44.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:22:44.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:22:45.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:45 smithi028 ceph-mon[24806]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:45.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:45 smithi028 ceph-mon[29568]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:45.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:45 smithi066 ceph-mon[25664]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:47.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:47 smithi028 ceph-mon[24806]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:47.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:47 smithi028 ceph-mon[29568]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:47.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:47 smithi066 ceph-mon[25664]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:49.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:49 smithi066 ceph-mon[25664]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:49.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:49 smithi028 ceph-mon[24806]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:49.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:49 smithi028 ceph-mon[29568]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:51.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:51 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:51.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:51 smithi028 ceph-mon[29568]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:51.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:51 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:51.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:51 smithi028 ceph-mon[24806]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:51.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:51 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:22:51.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:51 smithi066 ceph-mon[25664]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:52.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:22:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:52] "GET /metrics HTTP/1.1" 200 208015 "" "Prometheus/2.33.4" 2024-03-21T21:22:53.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:22:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:22:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:22:53.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:53 smithi028 ceph-mon[24806]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:53.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:53 smithi028 ceph-mon[29568]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:53.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:53 smithi066 ceph-mon[25664]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[24806]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[29568]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:22:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:55 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:22:55.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:55 smithi066 ceph-mon[25664]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:55.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:55 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:22:55.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:55 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:22:55.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:55 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:22:55.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:55 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:22:57.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:57 smithi066 ceph-mon[25664]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:57.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:57 smithi028 ceph-mon[24806]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:57.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:57 smithi028 ceph-mon[29568]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:22:59.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:22:59 smithi028 ceph-mon[24806]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:59.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:22:59 smithi028 ceph-mon[29568]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:22:59.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:22:59 smithi066 ceph-mon[25664]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:01.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:01 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:01.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:01 smithi028 ceph-mon[24806]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:01.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:01 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:01.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:01 smithi028 ceph-mon[29568]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:01.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:01 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:01.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:01 smithi066 ceph-mon[25664]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:02.941 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:23:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:02] "GET /metrics HTTP/1.1" 200 207993 "" "Prometheus/2.33.4" 2024-03-21T21:23:03.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:23:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:23:03.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:03 smithi028 ceph-mon[29568]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:03.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:03 smithi028 ceph-mon[24806]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:03.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:03 smithi066 ceph-mon[25664]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:05.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:05 smithi028 ceph-mon[29568]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:05.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:05 smithi028 ceph-mon[24806]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:05.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:05 smithi066 ceph-mon[25664]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:07.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:07 smithi028 ceph-mon[29568]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:07.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:07 smithi028 ceph-mon[24806]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:07.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:07 smithi066 ceph-mon[25664]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:09 smithi028 ceph-mon[29568]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:09.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:09 smithi028 ceph-mon[24806]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:09.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:09 smithi066 ceph-mon[25664]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:11.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:11 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:11.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:11 smithi028 ceph-mon[29568]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:11 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:11 smithi028 ceph-mon[24806]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:11.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:11 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:11.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:11 smithi066 ceph-mon[25664]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:12.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:23:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:12] "GET /metrics HTTP/1.1" 200 207993 "" "Prometheus/2.33.4" 2024-03-21T21:23:13.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:23:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:23:13.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:13 smithi028 ceph-mon[29568]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:13.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:13 smithi028 ceph-mon[24806]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:13.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:13 smithi066 ceph-mon[25664]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:15.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:15 smithi028 ceph-mon[29568]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:15 smithi028 ceph-mon[24806]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:15.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:15 smithi066 ceph-mon[25664]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:17.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:17 smithi028 ceph-mon[29568]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:17.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:17 smithi028 ceph-mon[24806]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:17.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:17 smithi066 ceph-mon[25664]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:19.495 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:19 smithi066 ceph-mon[25664]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:19.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:19 smithi028 ceph-mon[29568]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:19.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:19 smithi028 ceph-mon[24806]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:21.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:21 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:21.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:21 smithi028 ceph-mon[24806]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:21.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:21 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:21.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:21 smithi028 ceph-mon[29568]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:21.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:21 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:21.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:21 smithi066 ceph-mon[25664]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:22.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:23:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:22] "GET /metrics HTTP/1.1" 200 207946 "" "Prometheus/2.33.4" 2024-03-21T21:23:23.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:23:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:23:23.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:23 smithi028 ceph-mon[24806]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:23.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:23 smithi028 ceph-mon[29568]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:23.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:23 smithi066 ceph-mon[25664]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:25 smithi028 ceph-mon[24806]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:25.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:25 smithi028 ceph-mon[29568]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:25.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:25 smithi066 ceph-mon[25664]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:27.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:27 smithi028 ceph-mon[24806]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:27.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:27 smithi028 ceph-mon[29568]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:27.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:27 smithi066 ceph-mon[25664]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:29.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:29 smithi028 ceph-mon[24806]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:29.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:29 smithi028 ceph-mon[29568]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:29.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:29 smithi066 ceph-mon[25664]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:31.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:31 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:31.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:31 smithi028 ceph-mon[24806]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:31.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:31 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:31.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:31 smithi028 ceph-mon[29568]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:31.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:31 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:31.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:31 smithi066 ceph-mon[25664]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:32.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:23:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:32] "GET /metrics HTTP/1.1" 200 207951 "" "Prometheus/2.33.4" 2024-03-21T21:23:33.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:23:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:23:33.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:33 smithi028 ceph-mon[24806]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:33.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:33 smithi028 ceph-mon[29568]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:33.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:33 smithi066 ceph-mon[25664]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:35.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:35 smithi028 ceph-mon[24806]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:35.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:35 smithi028 ceph-mon[29568]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:35.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:35 smithi066 ceph-mon[25664]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:37.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:37 smithi028 ceph-mon[24806]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:37.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:37 smithi028 ceph-mon[29568]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:37.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:37 smithi066 ceph-mon[25664]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:39.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:39 smithi028 ceph-mon[24806]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:39.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:39 smithi028 ceph-mon[29568]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:39.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:39 smithi066 ceph-mon[25664]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:41.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-21T21:23:41.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:41 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:41.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:41 smithi028 ceph-mon[29568]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:41 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:41 smithi028 ceph-mon[24806]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:41.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:41 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:41.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:41 smithi066 ceph-mon[25664]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:42.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:23:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:42] "GET /metrics HTTP/1.1" 200 207951 "" "Prometheus/2.33.4" 2024-03-21T21:23:43.245 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:23:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:23:43.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:43 smithi028 ceph-mon[29568]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:43.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:43 smithi028 ceph-mon[24806]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:43.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:43 smithi066 ceph-mon[25664]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:44.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-21T21:23:44.666 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:23:44.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:23:44.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:23:44.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:23:44.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:23:44.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:23:44.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:23:44.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:23:44.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:23:44.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:23:44.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:23:44.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:23:45.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:45 smithi028 ceph-mon[24806]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:45.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:45 smithi028 ceph-mon[29568]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:45.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:45 smithi066 ceph-mon[25664]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:46.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-21T21:23:47.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:47 smithi028 ceph-mon[24806]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:47.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:47 smithi028 ceph-mon[29568]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:47.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:47 smithi066 ceph-mon[25664]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:49.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:23:49.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:49 smithi028 ceph-mon[24806]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:49.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:49 smithi028 ceph-mon[29568]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:49.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:49 smithi066 ceph-mon[25664]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:51.180 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (3m) 2m ago 4m 14.4M - ba2b418f427c 6d5934565000 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (3m) 2m ago 3m 45.2M - 8.3.5 dad864ee21e9 a550093f473a 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (3m) 2m ago 3m 68.2M - 3.5 e1d6a67b021e 173173f6d7ca 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e 2b9b0c91c423 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:9283 running (10m) 2m ago 10m 454M - 17.2.0 e1d6a67b021e 602bce1ea97f 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (10m) 2m ago 10m 50.1M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (8m) 2m ago 8m 41.7M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (9m) 2m ago 9m 39.7M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (4m) 2m ago 4m 16.5M - 1dbe0e931976 f01f375e6b14 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (4m) 2m ago 4m 16.0M - 1dbe0e931976 c03593861fb9 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (8m) 2m ago 8m 46.6M 4017M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (7m) 2m ago 7m 48.5M 4017M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (7m) 2m ago 7m 46.3M 4017M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (7m) 2m ago 7m 45.6M 4017M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:23:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (6m) 2m ago 6m 46.0M 4272M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:23:51.182 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (6m) 2m ago 6m 44.6M 4272M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:23:51.182 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (5m) 2m ago 5m 43.3M 4272M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:23:51.182 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (5m) 2m ago 5m 44.8M 4272M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:23:51.182 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (3m) 2m ago 4m 45.0M - 514e6a882f6e c61488efcefc 2024-03-21T21:23:51.182 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (3m) 2m ago 3m 70.4M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:23:51.182 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (3m) 2m ago 3m 68.9M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:23:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:51 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:51 smithi028 ceph-mon[24806]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:51.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:51 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:51.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:51 smithi028 ceph-mon[29568]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:51.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:51 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:23:51.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:51 smithi066 ceph-mon[25664]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:51.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions' 2024-03-21T21:23:52.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:52 smithi028 ceph-mon[29568]: from='client.24895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:23:52.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:52 smithi028 ceph-mon[24806]: from='client.24895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:23:52.692 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:23:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:52] "GET /metrics HTTP/1.1" 200 207946 "" "Prometheus/2.33.4" 2024-03-21T21:23:52.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:52 smithi066 ceph-mon[25664]: from='client.24895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:23:53.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:23:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:23:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:23:53.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:53 smithi028 ceph-mon[24806]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:53.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:53 smithi028 ceph-mon[29568]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:53.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:53 smithi066 ceph-mon[25664]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:23:53.949 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:23:53.949 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:23:53.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:23:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:23:54.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:54 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/4024835038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:23:54.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:54 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/4024835038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:23:54.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph -s' 2024-03-21T21:23:54.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:54 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/4024835038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:23:55.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:55 smithi066 ceph-mon[25664]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:55.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:55 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:23:55.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:55 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:23:55.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:55 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:23:55.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:55 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:23:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[29568]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:23:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:23:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:23:55.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:23:55.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[24806]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:55.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:23:55.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:23:55.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:23:55.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:55 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:23:56.744 INFO:teuthology.orchestra.run.smithi028.stdout: cluster: 2024-03-21T21:23:56.744 INFO:teuthology.orchestra.run.smithi028.stdout: id: c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: health: HEALTH_OK 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: services: 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: mgr: y(active, since 5m), standbys: x 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: data: 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: pools: 6 pools, 161 pgs 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: objects: 209 objects, 585 KiB 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: usage: 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: pgs: 161 active+clean 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:23:56.745 INFO:teuthology.orchestra.run.smithi028.stdout: io: 2024-03-21T21:23:56.746 INFO:teuthology.orchestra.run.smithi028.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-21T21:23:56.746 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:23:57.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:57 smithi028 ceph-mon[24806]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:57.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:57 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2600553880' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:23:57.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:57 smithi028 ceph-mon[29568]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:57.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:57 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2600553880' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:23:57.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ls' 2024-03-21T21:23:57.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:57 smithi066 ceph-mon[25664]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:23:57.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:57 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2600553880' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:23:59.482 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-21T21:23:59.482 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi028=a;count:1 2024-03-21T21:23:59.482 INFO:teuthology.orchestra.run.smithi028.stdout:grafana ?:3000 1/1 3m ago 5m smithi066=a;count:1 2024-03-21T21:23:59.482 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-03-21T21:23:59.482 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 2/2 3m ago 8m smithi028=y;smithi066=x;count:2 2024-03-21T21:23:59.483 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 3m ago 9m smithi028:172.21.15.28=a;smithi028:[v2:172.21.15.28:3301,v1:172.21.15.28:6790]=c;smithi066:172.21.15.66=b;count:3 2024-03-21T21:23:59.483 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi028=a;smithi066=b;count:2 2024-03-21T21:23:59.483 INFO:teuthology.orchestra.run.smithi028.stdout:osd 8 3m ago - 2024-03-21T21:23:59.483 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus ?:9095 1/1 3m ago 5m smithi066=a;count:1 2024-03-21T21:23:59.483 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-03-21T21:23:59.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:23:59 smithi028 ceph-mon[24806]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:59.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:23:59 smithi028 ceph-mon[29568]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:23:59.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:23:59 smithi066 ceph-mon[25664]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:00.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-21T21:24:00.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:00 smithi028 ceph-mon[29568]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:24:00.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:00 smithi028 ceph-mon[24806]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:24:00.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:00 smithi066 ceph-mon[25664]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:24:01.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:01 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:01.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:01 smithi028 ceph-mon[24806]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:01.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:01 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:01.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:01 smithi028 ceph-mon[29568]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:01.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:01 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:01.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:01 smithi066 ceph-mon[25664]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:02.546 INFO:teuthology.orchestra.run.smithi028.stdout:Scheduled to redeploy mgr.x on host 'smithi066' 2024-03-21T21:24:02.547 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:02 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/200710549' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T21:24:02.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:24:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:02] "GET /metrics HTTP/1.1" 200 207937 "" "Prometheus/2.33.4" 2024-03-21T21:24:02.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:02 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/200710549' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T21:24:02.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:02 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/200710549' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T21:24:03.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:24:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:24:03.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps --refresh' 2024-03-21T21:24:03.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[29568]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:03.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[29568]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:24:03.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:03.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[29568]: Schedule redeploy daemon mgr.x 2024-03-21T21:24:03.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:24:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[24806]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[24806]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:24:03.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:03.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[24806]: Schedule redeploy daemon mgr.x 2024-03-21T21:24:03.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:03 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:24:03.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:03 smithi066 ceph-mon[25664]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:03.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:03 smithi066 ceph-mon[25664]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:24:03.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:03 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:03.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:03 smithi066 ceph-mon[25664]: Schedule redeploy daemon mgr.x 2024-03-21T21:24:03.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:03 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:24:05.305 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (3m) 3m ago 4m 14.4M - ba2b418f427c 6d5934565000 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (3m) 3m ago 3m 45.2M - 8.3.5 dad864ee21e9 a550093f473a 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (3m) 3m ago 3m 68.2M - 3.5 e1d6a67b021e 173173f6d7ca 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e 2b9b0c91c423 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:9283 running (10m) 3m ago 10m 454M - 17.2.0 e1d6a67b021e 602bce1ea97f 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (10m) 3m ago 10m 50.1M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (9m) 3m ago 9m 41.7M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (9m) 3m ago 9m 39.7M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (4m) 3m ago 4m 16.5M - 1dbe0e931976 f01f375e6b14 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (4m) 3m ago 4m 16.0M - 1dbe0e931976 c03593861fb9 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (8m) 3m ago 8m 46.6M 4017M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (8m) 3m ago 8m 48.5M 4017M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:24:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (7m) 3m ago 7m 46.3M 4017M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:24:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (7m) 3m ago 7m 45.6M 4017M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:24:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (6m) 3m ago 6m 46.0M 4272M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:24:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (6m) 3m ago 6m 44.6M 4272M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:24:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (6m) 3m ago 6m 43.3M 4272M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:24:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (5m) 3m ago 5m 44.8M 4272M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:24:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (3m) 3m ago 5m 45.0M - 514e6a882f6e c61488efcefc 2024-03-21T21:24:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (3m) 3m ago 3m 70.4M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:24:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (3m) 3m ago 3m 68.9M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:24:05.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:05 smithi028 ceph-mon[29568]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:05.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:05 smithi028 ceph-mon[24806]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:05.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:05 smithi066 ceph-mon[25664]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:06.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'sleep 180' 2024-03-21T21:24:06.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:06 smithi028 ceph-mon[29568]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:24:06.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:06 smithi028 ceph-mon[24806]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:24:06.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:06 smithi066 ceph-mon[25664]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:24:07.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:07 smithi028 ceph-mon[24806]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:07.896 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:07 smithi028 ceph-mon[29568]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:07.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:07 smithi066 ceph-mon[25664]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:09.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:09 smithi028 ceph-mon[24806]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:09.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:09 smithi028 ceph-mon[29568]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:09.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:09 smithi066 ceph-mon[25664]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:11.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:11 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:11.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:11 smithi028 ceph-mon[29568]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:11 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:11 smithi028 ceph-mon[24806]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:11.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:11 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:11.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:11 smithi066 ceph-mon[25664]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:12.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:24:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:12] "GET /metrics HTTP/1.1" 200 207937 "" "Prometheus/2.33.4" 2024-03-21T21:24:13.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:24:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:24:13.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:13 smithi028 ceph-mon[29568]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:13.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:13 smithi028 ceph-mon[24806]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:13.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:13 smithi066 ceph-mon[25664]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:15.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:15 smithi028 ceph-mon[29568]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:15 smithi028 ceph-mon[24806]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:15.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:15 smithi066 ceph-mon[25664]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:17.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:17 smithi028 ceph-mon[29568]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:17.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:17 smithi028 ceph-mon[24806]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:17.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:17 smithi066 ceph-mon[25664]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:19.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:19 smithi028 ceph-mon[29568]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:19.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:19 smithi028 ceph-mon[24806]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:19.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:19 smithi066 ceph-mon[25664]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:21.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:21 smithi028 ceph-mon[29568]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:21.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:21 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:21.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:21 smithi028 ceph-mon[24806]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:21.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:21 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:21.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:21 smithi066 ceph-mon[25664]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:21.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:21 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:22.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:24:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:22] "GET /metrics HTTP/1.1" 200 207942 "" "Prometheus/2.33.4" 2024-03-21T21:24:23.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:24:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:24:23.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:23 smithi028 ceph-mon[29568]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:23.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:23 smithi028 ceph-mon[24806]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:23.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:23 smithi066 ceph-mon[25664]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:25.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:25 smithi028 ceph-mon[24806]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:25.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:25 smithi028 ceph-mon[29568]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:25.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:25 smithi066 ceph-mon[25664]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:27.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:27 smithi028 ceph-mon[24806]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:27.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:27 smithi028 ceph-mon[29568]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:27.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:27 smithi066 ceph-mon[25664]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:29.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:29 smithi028 ceph-mon[29568]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:29.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:29 smithi028 ceph-mon[24806]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:29.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:29 smithi066 ceph-mon[25664]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:31.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:31 smithi028 ceph-mon[24806]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:31.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:31 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:31.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:31 smithi028 ceph-mon[29568]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:31.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:31 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:31.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:31 smithi066 ceph-mon[25664]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:31.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:31 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:32.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:24:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:32] "GET /metrics HTTP/1.1" 200 207951 "" "Prometheus/2.33.4" 2024-03-21T21:24:33.221 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:24:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:24:33.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:33.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:24:33.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:24:33.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[29568]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:33.341 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:33.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:33.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:24:33.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:24:33.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[24806]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:33.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:33 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:33.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:33 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:33.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:33 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:24:33.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:33 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:24:33.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:33 smithi066 ceph-mon[25664]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:33.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:33 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:35.496 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:35 smithi066 ceph-mon[25664]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:35.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:35 smithi028 ceph-mon[24806]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:35.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:35 smithi028 ceph-mon[29568]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:36.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:24:36.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:24:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:24:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:36 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:24:36.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:36 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:36.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:36 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:36.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:36 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:24:36.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:36 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:24:36.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:36 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:24:36.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:36 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:24:36.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:36 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:24:37.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:37 smithi028 ceph-mon[24806]: Deploying daemon mgr.x on smithi066 2024-03-21T21:24:37.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:37 smithi028 ceph-mon[24806]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:37.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:37 smithi028 ceph-mon[29568]: Deploying daemon mgr.x on smithi066 2024-03-21T21:24:37.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:37 smithi028 ceph-mon[29568]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:37.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:37 smithi066 ceph-mon[25664]: Deploying daemon mgr.x on smithi066 2024-03-21T21:24:37.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:37 smithi066 ceph-mon[25664]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:39.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:39 smithi028 ceph-mon[24806]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:39.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:39 smithi028 ceph-mon[29568]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:39.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:39 smithi066 ceph-mon[25664]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:41.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:41 smithi028 ceph-mon[29568]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:41.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:41 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:41.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:41 smithi028 ceph-mon[24806]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:41.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:41 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:41.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:41 smithi066 ceph-mon[25664]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:41.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:41 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:42.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:24:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:42] "GET /metrics HTTP/1.1" 200 207951 "" "Prometheus/2.33.4" 2024-03-21T21:24:43.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:24:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:24:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:43 smithi028 ceph-mon[24806]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:43 smithi028 ceph-mon[29568]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:43.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:24:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:24:43.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:24:43.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:24:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:24:43.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:24:43.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:43 smithi066 ceph-mon[25664]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:44.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:24:44.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:24:44.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:24:44.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:24:44.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:24:44.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:24:44.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:24:44.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:24:44.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:24:44.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:24:44.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:24:44.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:24:45.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:45 smithi028 ceph-mon[24806]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:45.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:45 smithi028 ceph-mon[29568]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:45.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:45 smithi066 ceph-mon[25664]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:47.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:47 smithi028 ceph-mon[24806]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:47.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:47 smithi028 ceph-mon[29568]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:47.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:47 smithi066 ceph-mon[25664]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:49.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:49 smithi028 ceph-mon[24806]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:49.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:49 smithi028 ceph-mon[29568]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:49.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:49 smithi066 ceph-mon[25664]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:51.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:51 smithi028 ceph-mon[24806]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:51.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:51 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:51.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:51 smithi028 ceph-mon[29568]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:51.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:51 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:51.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:51 smithi066 ceph-mon[25664]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:51.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:51 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:24:52.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:24:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:52] "GET /metrics HTTP/1.1" 200 207960 "" "Prometheus/2.33.4" 2024-03-21T21:24:53.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:24:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:24:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:24:53.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:53 smithi028 ceph-mon[24806]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:53.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:53 smithi028 ceph-mon[29568]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:24:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:24:53.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:24:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:24:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:24:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:24:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:24:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:24:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:24:53.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:53 smithi066 ceph-mon[25664]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:55.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:55 smithi028 ceph-mon[24806]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:55.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:55 smithi028 ceph-mon[29568]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:55.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:55 smithi066 ceph-mon[25664]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:57.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:57 smithi028 ceph-mon[29568]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:57.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:57 smithi028 ceph-mon[24806]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:57.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:57 smithi066 ceph-mon[25664]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:24:59.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:24:59 smithi028 ceph-mon[24806]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:59.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:24:59 smithi028 ceph-mon[29568]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:24:59.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:24:59 smithi066 ceph-mon[25664]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:01.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:01 smithi028 ceph-mon[29568]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:01.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:01 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:01.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:01 smithi028 ceph-mon[24806]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:01.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:01 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:01.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:01 smithi066 ceph-mon[25664]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:01.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:01 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:02.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:25:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:02] "GET /metrics HTTP/1.1" 200 207961 "" "Prometheus/2.33.4" 2024-03-21T21:25:03.188 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[26865]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:25:03.476 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:03 smithi066 ceph-mon[25664]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:03.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:03 smithi028 ceph-mon[24806]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:03.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:03 smithi028 ceph-mon[29568]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:03.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:25:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:03.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:03.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:25:04.884 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:04 smithi066 systemd[1]: Stopping Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:25:04.884 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:04 smithi066 bash[45764]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr.x" found: no such container 2024-03-21T21:25:04.884 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:04 smithi066 podman[45777]: 2024-03-21 21:25:04.666125896 +0000 UTC m=+0.127270794 container died 2b9b0c91c423b80f62c9ecf7c00299e06ce7910aa2f98dbd9df8114be092fe95 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, ceph=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, version=8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=) 2024-03-21T21:25:05.193 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:04 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:04.827Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:25:05.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:04 smithi066 podman[45777]: 2024-03-21 21:25:04.977732616 +0000 UTC m=+0.438877506 container cleanup 2b9b0c91c423b80f62c9ecf7c00299e06ce7910aa2f98dbd9df8114be092fe95 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-type=git, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, name=centos-stream, architecture=x86_64, GIT_CLEAN=True, version=8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-21T21:25:05.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:04 smithi066 bash[45777]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x 2024-03-21T21:25:05.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 bash[45803]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr.x" found: no such container 2024-03-21T21:25:05.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:05 smithi028 ceph-mon[24806]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:05.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:05 smithi028 ceph-mon[29568]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:05.717 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:05 smithi066 ceph-mon[25664]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:05.717 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 podman[45790]: 2024-03-21 21:25:05.288000119 +0000 UTC m=+0.617460218 container remove 2b9b0c91c423b80f62c9ecf7c00299e06ce7910aa2f98dbd9df8114be092fe95 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, name=centos-stream, io.k8s.display-name=CentOS Stream 8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-21T21:25:05.718 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:25:05.718 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-21T21:25:05.718 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 systemd[1]: Stopped Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:25:05.718 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service: Consumed 17.816s CPU time. 2024-03-21T21:25:05.718 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 systemd[1]: Starting Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:25:05.980 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 podman[45905]: 2024-03-21 21:25:05.625042204 +0000 UTC m=+0.019703992 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:25:05.980 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 podman[45905]: 2024-03-21 21:25:05.751511685 +0000 UTC m=+0.146174018 container create 7fe31a1b09b61c385055e587f9e2de0812ae2c00577b58b78cb1be31e3f71f73 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-21T21:25:06.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 podman[45905]: 2024-03-21 21:25:05.983673209 +0000 UTC m=+0.378334991 container init 7fe31a1b09b61c385055e587f9e2de0812ae2c00577b58b78cb1be31e3f71f73 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-21T21:25:06.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:05 smithi066 podman[45905]: 2024-03-21 21:25:05.986847175 +0000 UTC m=+0.381508964 container start 7fe31a1b09b61c385055e587f9e2de0812ae2c00577b58b78cb1be31e3f71f73 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:25:06.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:06 smithi066 ceph-mgr[45923]: -- 172.21.15.66:0/2701398900 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5607ffdee5a0 con 0x5607ffdc3000 2024-03-21T21:25:06.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:06 smithi066 bash[45905]: 7fe31a1b09b61c385055e587f9e2de0812ae2c00577b58b78cb1be31e3f71f73 2024-03-21T21:25:06.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:06 smithi066 systemd[1]: Started Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:25:06.996 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:06 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:06.575+0000 7f8f644f9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:25:06.996 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:06 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:06.675+0000 7f8f644f9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:25:07.390 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:07.390 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:07.390 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:25:07.390 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:25:07.390 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:25:07.390 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-mon[25664]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:07.390 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:07.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:07.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:07.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:25:07.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:25:07.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:25:07.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[29568]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:07.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:07.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:07.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:07.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:25:07.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:25:07.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:25:07.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[24806]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:07.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:07 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:07.644 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:07.392+0000 7f8f644f9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:25:07.644 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:07.491+0000 7f8f644f9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:25:07.961 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:07.646+0000 7f8f644f9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:25:07.961 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:07.748+0000 7f8f644f9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:25:07.961 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:07.836+0000 7f8f644f9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:25:08.233 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:08.122+0000 7f8f644f9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T21:25:08.496 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:08.236+0000 7f8f644f9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:25:08.496 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:08.338+0000 7f8f644f9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:25:08.496 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:08.447+0000 7f8f644f9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:25:08.894 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:08.645+0000 7f8f644f9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:25:09.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:09 smithi028 ceph-mon[29568]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:09.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:09 smithi028 ceph-mon[24806]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:09.585 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:09 smithi066 ceph-mon[25664]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:09.845 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:09.711+0000 7f8f644f9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:25:10.189 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:09.848+0000 7f8f644f9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:25:10.496 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:10.217+0000 7f8f644f9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:25:10.496 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:10.322+0000 7f8f644f9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:25:10.996 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:10.529+0000 7f8f644f9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:25:10.996 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:10.743+0000 7f8f644f9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:25:11.323 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:11.055+0000 7f8f644f9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:25:11.323 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:11.326+0000 7f8f644f9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:25:11.667 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:11.574+0000 7f8f644f9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:25:11.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:11 smithi028 ceph-mon[24806]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:11.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:11 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:11.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:11 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:11.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:11 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:11.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:11 smithi028 ceph-mon[29568]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:11.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:11 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:11.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:11 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:11.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:11 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:11.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:11 smithi066 ceph-mon[25664]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:11.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:11 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:11.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:11 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:11.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:11 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:25:12.746 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:12.423+0000 7f8f644f9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:25:12.942 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:25:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:12] "GET /metrics HTTP/1.1" 200 207961 "" "Prometheus/2.33.4" 2024-03-21T21:25:13.351 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:13.023+0000 7f8f644f9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:25:13.352 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:13.126+0000 7f8f644f9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:25:13.669 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:13.354+0000 7f8f644f9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:25:13.669 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:25:13] ENGINE Bus STARTING 2024-03-21T21:25:13.669 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: CherryPy Checker: 2024-03-21T21:25:13.669 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: The Application mounted at '' has an empty config. 2024-03-21T21:25:13.670 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:25:13.670 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:25:13] ENGINE Serving on http://:::9283 2024-03-21T21:25:13.670 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:25:13] ENGINE Bus STARTED 2024-03-21T21:25:13.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[24806]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:13.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[24806]: Standby manager daemon x restarted 2024-03-21T21:25:13.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[24806]: Standby manager daemon x started 2024-03-21T21:25:13.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:25:13.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:25:13.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:25:13.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:25:13.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[29568]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:13.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[29568]: Standby manager daemon x restarted 2024-03-21T21:25:13.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[29568]: Standby manager daemon x started 2024-03-21T21:25:13.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:25:13.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:25:13.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:25:13.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:25:13.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:25:13.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:25:13.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:25:13.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:13.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-mon[25664]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:13.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-mon[25664]: Standby manager daemon x restarted 2024-03-21T21:25:13.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-mon[25664]: Standby manager daemon x started 2024-03-21T21:25:13.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:25:13.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:25:13.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:25:13.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:13 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:25:15.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:15 smithi028 ceph-mon[24806]: mgrmap e21: y(active, since 6m), standbys: x 2024-03-21T21:25:15.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:15 smithi028 ceph-mon[24806]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:15.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:15 smithi028 ceph-mon[29568]: mgrmap e21: y(active, since 6m), standbys: x 2024-03-21T21:25:15.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:15 smithi028 ceph-mon[29568]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:15.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:15 smithi066 ceph-mon[25664]: mgrmap e21: y(active, since 6m), standbys: x 2024-03-21T21:25:15.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:15 smithi066 ceph-mon[25664]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:17.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:17 smithi028 ceph-mon[24806]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:17.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:17 smithi028 ceph-mon[29568]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:17.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:17 smithi066 ceph-mon[25664]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:19.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:19 smithi028 ceph-mon[29568]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:19.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:19 smithi028 ceph-mon[24806]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:19.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:19 smithi066 ceph-mon[25664]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:21.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:21 smithi028 ceph-mon[29568]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:21.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:21 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:21.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:21 smithi028 ceph-mon[24806]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:21.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:21 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:21.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:21 smithi066 ceph-mon[25664]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:21.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:21 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:22.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:25:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:22] "GET /metrics HTTP/1.1" 200 207952 "" "Prometheus/2.33.4" 2024-03-21T21:25:23.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:25:23.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:23 smithi028 ceph-mon[29568]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:23.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:23 smithi028 ceph-mon[24806]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:23.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:25:23.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:23.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:23.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:25:23.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:23 smithi066 ceph-mon[25664]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:25.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:25 smithi028 ceph-mon[24806]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:25.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:25 smithi028 ceph-mon[29568]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:25.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:25 smithi066 ceph-mon[25664]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:27.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:27 smithi028 ceph-mon[24806]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:27.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:27 smithi028 ceph-mon[29568]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:27.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:27 smithi066 ceph-mon[25664]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:29.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:29 smithi028 ceph-mon[24806]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:29.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:29 smithi028 ceph-mon[29568]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:29.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:29 smithi066 ceph-mon[25664]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:31.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:31 smithi028 ceph-mon[24806]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:31.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:31 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:31.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:31 smithi028 ceph-mon[29568]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:31.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:31 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:31.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:31 smithi066 ceph-mon[25664]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:31.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:31 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:32.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:25:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:32] "GET /metrics HTTP/1.1" 200 207967 "" "Prometheus/2.33.4" 2024-03-21T21:25:33.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:25:33.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:33 smithi028 ceph-mon[24806]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:33.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:33 smithi028 ceph-mon[29568]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:33.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:25:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:25:33.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:33.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:25:33.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:33 smithi066 ceph-mon[25664]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:35.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:35 smithi028 ceph-mon[24806]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:35.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:35 smithi028 ceph-mon[29568]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:35.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:35 smithi066 ceph-mon[25664]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:37.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:37 smithi028 ceph-mon[24806]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:37.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:37 smithi028 ceph-mon[29568]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:37.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:37 smithi066 ceph-mon[25664]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:39.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:39 smithi028 ceph-mon[24806]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:39.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:39 smithi028 ceph-mon[29568]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:39.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:39 smithi066 ceph-mon[25664]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:41.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:41 smithi028 ceph-mon[24806]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:41.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:41 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:41.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:41 smithi028 ceph-mon[29568]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:41.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:41 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:41.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:41 smithi066 ceph-mon[25664]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:41.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:41 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:42.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:25:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:42] "GET /metrics HTTP/1.1" 200 207967 "" "Prometheus/2.33.4" 2024-03-21T21:25:43.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:25:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:43 smithi028 ceph-mon[24806]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:43 smithi028 ceph-mon[29568]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:43.693 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:25:43.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:43.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:25:43.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:43.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:43 smithi066 ceph-mon[25664]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:44.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:25:44.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:25:44.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:25:44.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:25:44.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:25:44.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:25:44.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:25:44.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:25:44.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:25:44.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:25:44.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:25:44.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:25:45.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:45 smithi028 ceph-mon[24806]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:45.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:45 smithi028 ceph-mon[29568]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:45.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:45 smithi066 ceph-mon[25664]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:47.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:47 smithi028 ceph-mon[24806]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:47.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:47 smithi028 ceph-mon[29568]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:47.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:47 smithi066 ceph-mon[25664]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:49.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:49 smithi028 ceph-mon[24806]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:49.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:49 smithi028 ceph-mon[29568]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:49.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:49 smithi066 ceph-mon[25664]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:51.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:51 smithi028 ceph-mon[24806]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:51.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:51 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:51.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:51 smithi028 ceph-mon[29568]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:51.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:51 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:51.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:51 smithi066 ceph-mon[25664]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:51.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:51 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:25:52.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:25:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:52] "GET /metrics HTTP/1.1" 200 207959 "" "Prometheus/2.33.4" 2024-03-21T21:25:53.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:25:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:25:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:25:53.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:53 smithi028 ceph-mon[29568]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:53.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:53 smithi028 ceph-mon[24806]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:25:53.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:25:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:25:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:25:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:25:53.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:53 smithi066 ceph-mon[25664]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:55.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:55 smithi028 ceph-mon[29568]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:55.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:55 smithi028 ceph-mon[24806]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:55.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:55 smithi066 ceph-mon[25664]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:57.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:57 smithi028 ceph-mon[29568]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:57.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:57 smithi028 ceph-mon[24806]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:57.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:57 smithi066 ceph-mon[25664]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:25:59.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:25:59 smithi028 ceph-mon[29568]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:59.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:25:59 smithi028 ceph-mon[24806]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:25:59.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:25:59 smithi066 ceph-mon[25664]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:01.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:01 smithi028 ceph-mon[24806]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:01.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:01 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:01.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:01 smithi028 ceph-mon[29568]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:01.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:01 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:01.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:01 smithi066 ceph-mon[25664]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:01.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:01 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:02.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:26:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:02] "GET /metrics HTTP/1.1" 200 207962 "" "Prometheus/2.33.4" 2024-03-21T21:26:03.246 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:26:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:26:03.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:03 smithi028 ceph-mon[24806]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:03.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:03 smithi028 ceph-mon[29568]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:03.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:26:03.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:26:03.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:26:03.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:03.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:26:03.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:03 smithi066 ceph-mon[25664]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:05.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:05 smithi028 ceph-mon[24806]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:05.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:05 smithi028 ceph-mon[29568]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:05.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:05 smithi066 ceph-mon[25664]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:07.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:07 smithi066 ceph-mon[25664]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:07.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:07 smithi028 ceph-mon[24806]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:07.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:07 smithi028 ceph-mon[29568]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:09.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:09 smithi028 ceph-mon[24806]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:09.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:09 smithi028 ceph-mon[29568]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:09.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:09 smithi066 ceph-mon[25664]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[24806]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:26:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:26:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:26:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[29568]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:26:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:26:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:11 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:26:11.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:11 smithi066 ceph-mon[25664]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:11.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:11 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:11.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:11 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:26:11.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:11 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:26:11.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:11 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:26:12.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:12 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:26:12.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:12 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:26:12.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:12 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:26:12.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:26:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:12] "GET /metrics HTTP/1.1" 200 207962 "" "Prometheus/2.33.4" 2024-03-21T21:26:13.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:26:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:26:13.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:13 smithi066 ceph-mon[25664]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:13.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:13 smithi028 ceph-mon[24806]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:13.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:13 smithi028 ceph-mon[29568]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:13.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:26:13.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:26:13.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:13.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:26:13.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:13.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:26:15.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:15 smithi066 ceph-mon[25664]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:15.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:15 smithi028 ceph-mon[24806]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:15.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:15 smithi028 ceph-mon[29568]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:17.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:17 smithi066 ceph-mon[25664]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:17.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:17 smithi028 ceph-mon[24806]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:17.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:17 smithi028 ceph-mon[29568]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:19.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:19 smithi028 ceph-mon[24806]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:19.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:19 smithi028 ceph-mon[29568]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:19.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:19 smithi066 ceph-mon[25664]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:21.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:21 smithi028 ceph-mon[24806]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:21.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:21 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:21.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:21 smithi028 ceph-mon[29568]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:21.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:21 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:21.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:21 smithi066 ceph-mon[25664]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:21.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:21 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:22.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:26:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:22] "GET /metrics HTTP/1.1" 200 207964 "" "Prometheus/2.33.4" 2024-03-21T21:26:23.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:26:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:26:23.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:23 smithi028 ceph-mon[24806]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:23.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:23 smithi028 ceph-mon[29568]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:23.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:26:23.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:26:23.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:23.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:26:23.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:23.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:26:23.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:23 smithi066 ceph-mon[25664]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:25.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:25 smithi028 ceph-mon[29568]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:25.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:25 smithi028 ceph-mon[24806]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:25.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:25 smithi066 ceph-mon[25664]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:27.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:27 smithi028 ceph-mon[24806]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:27.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:27 smithi028 ceph-mon[29568]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:27.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:27 smithi066 ceph-mon[25664]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:29.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:29 smithi028 ceph-mon[29568]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:29.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:29 smithi028 ceph-mon[24806]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:29.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:29 smithi066 ceph-mon[25664]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:31.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:31 smithi028 ceph-mon[29568]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:31.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:31 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:31.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:31 smithi028 ceph-mon[24806]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:31.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:31 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:31.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:31 smithi066 ceph-mon[25664]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:31.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:31 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:32.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:26:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:32] "GET /metrics HTTP/1.1" 200 207958 "" "Prometheus/2.33.4" 2024-03-21T21:26:33.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:26:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:26:33.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:33 smithi028 ceph-mon[24806]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:33.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:33 smithi028 ceph-mon[29568]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:33.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:26:33.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:26:33.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:33.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:26:33.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:33.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:26:33.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:33 smithi066 ceph-mon[25664]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:35.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:35 smithi028 ceph-mon[24806]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:35.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:35 smithi028 ceph-mon[29568]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:35.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:35 smithi066 ceph-mon[25664]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:37.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:37 smithi028 ceph-mon[24806]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:37.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:37 smithi028 ceph-mon[29568]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:37.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:37 smithi066 ceph-mon[25664]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:39.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:39 smithi028 ceph-mon[24806]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:39.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:39 smithi028 ceph-mon[29568]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:39.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:39 smithi066 ceph-mon[25664]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:41.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:41 smithi028 ceph-mon[29568]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:41.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:41 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:41.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:41 smithi028 ceph-mon[24806]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:41.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:41 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:41.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:41 smithi066 ceph-mon[25664]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:41.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:41 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:42.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:26:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:42] "GET /metrics HTTP/1.1" 200 207958 "" "Prometheus/2.33.4" 2024-03-21T21:26:43.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:26:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:26:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:43 smithi028 ceph-mon[29568]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:43 smithi028 ceph-mon[24806]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:43.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:26:43.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:26:43.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:26:43.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:43.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:26:43.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:43 smithi066 ceph-mon[25664]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:44.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:26:44.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:26:44.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:44 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:26:44.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:44 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:26:44.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:26:44.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:26:44.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:44 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:26:44.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:44 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:26:44.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:26:44.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:26:44.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:44 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:26:44.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:44 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:26:45.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:45 smithi028 ceph-mon[24806]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:45.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:45 smithi028 ceph-mon[29568]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:45.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:45 smithi066 ceph-mon[25664]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:47.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:47 smithi028 ceph-mon[24806]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:47.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:47 smithi028 ceph-mon[29568]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:47.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:47 smithi066 ceph-mon[25664]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:49.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:49 smithi028 ceph-mon[24806]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:49.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:49 smithi028 ceph-mon[29568]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:49.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:49 smithi066 ceph-mon[25664]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:51.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:51 smithi028 ceph-mon[24806]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:51.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:51 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:51.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:51 smithi028 ceph-mon[29568]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:51.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:51 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:51.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:51 smithi066 ceph-mon[25664]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:51.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:51 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:26:52.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:26:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:52] "GET /metrics HTTP/1.1" 200 207960 "" "Prometheus/2.33.4" 2024-03-21T21:26:53.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:26:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:26:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:26:53.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:53 smithi028 ceph-mon[24806]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:53.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:53 smithi028 ceph-mon[29568]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:26:53.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:26:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:26:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:26:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:26:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:26:53.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:53 smithi066 ceph-mon[25664]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:55.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:55 smithi028 ceph-mon[24806]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:55.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:55 smithi028 ceph-mon[29568]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:55.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:55 smithi066 ceph-mon[25664]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:57.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:57 smithi028 ceph-mon[24806]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:57.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:57 smithi028 ceph-mon[29568]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:57.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:57 smithi066 ceph-mon[25664]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:26:59.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:26:59 smithi028 ceph-mon[29568]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:59.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:26:59 smithi028 ceph-mon[24806]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:26:59.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:26:59 smithi066 ceph-mon[25664]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:01.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:01 smithi028 ceph-mon[24806]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:01.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:01 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:01.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:01 smithi028 ceph-mon[29568]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:01.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:01 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:01.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:01 smithi066 ceph-mon[25664]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:01.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:01 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:02.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:02] "GET /metrics HTTP/1.1" 200 207961 "" "Prometheus/2.33.4" 2024-03-21T21:27:03.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:27:03.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:03 smithi028 ceph-mon[24806]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:03.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:03 smithi028 ceph-mon[29568]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:03.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:27:03.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:03.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:03.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:03.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:03.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:27:03.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:03 smithi066 ceph-mon[25664]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:05.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:05 smithi028 ceph-mon[24806]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:05.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:05 smithi028 ceph-mon[29568]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:05.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:05 smithi066 ceph-mon[25664]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:07.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:07 smithi028 ceph-mon[24806]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:07.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:07 smithi028 ceph-mon[29568]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:07.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:07 smithi066 ceph-mon[25664]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:08.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:27:09.551 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:09 smithi028 ceph-mon[29568]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:09.551 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:09 smithi028 ceph-mon[24806]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:09.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:09 smithi066 ceph-mon[25664]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:10.889 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (6m) 2m ago 7m 24.9M - ba2b418f427c 6d5934565000 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (7m) 2m ago 7m 53.9M - 8.3.5 dad864ee21e9 a550093f473a 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (6m) 2m ago 6m 48.8M - 3.5 e1d6a67b021e 173173f6d7ca 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283 running (2m) 2m ago 11m 76.3M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 7fe31a1b09b6 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:9283 running (13m) 2m ago 13m 459M - 17.2.0 e1d6a67b021e 602bce1ea97f 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (13m) 2m ago 13m 49.6M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (12m) 2m ago 12m 42.7M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (12m) 2m ago 12m 40.5M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (8m) 2m ago 8m 15.1M - 1dbe0e931976 f01f375e6b14 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (8m) 2m ago 8m 18.8M - 1dbe0e931976 c03593861fb9 2024-03-21T21:27:10.890 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (11m) 2m ago 11m 49.5M 4017M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (11m) 2m ago 11m 50.2M 4017M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (10m) 2m ago 10m 48.2M 4017M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (10m) 2m ago 10m 48.7M 4017M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (10m) 2m ago 10m 49.2M 4272M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (9m) 2m ago 9m 48.0M 4272M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (9m) 2m ago 9m 46.4M 4272M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (8m) 2m ago 8m 48.1M 4272M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (6m) 2m ago 8m 67.7M - 514e6a882f6e c61488efcefc 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (6m) 2m ago 6m 71.4M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:27:10.891 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (6m) 2m ago 6m 70.0M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:27:11.512 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:11 smithi028 ceph-mon[29568]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:11.512 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:11 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:11.512 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:11 smithi028 ceph-mon[29568]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:27:11.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:11 smithi028 ceph-mon[24806]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:11.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:11 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:11.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:11 smithi028 ceph-mon[24806]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:27:11.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions' 2024-03-21T21:27:11.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:11 smithi066 ceph-mon[25664]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:11.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:11 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:11.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:11 smithi066 ceph-mon[25664]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:27:12.664 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:12] "GET /metrics HTTP/1.1" 200 207961 "" "Prometheus/2.33.4" 2024-03-21T21:27:12.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:12 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:27:12.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:12 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:27:12.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:12 smithi028 ceph-mon[24806]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:27:12.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:12 smithi028 ceph-mon[24806]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:27:12.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:12 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:27:12.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:12 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:27:12.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:12 smithi028 ceph-mon[29568]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:27:12.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:12 smithi028 ceph-mon[29568]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:27:12.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:12 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:27:12.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:12 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:27:12.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:12 smithi066 ceph-mon[25664]: from='mgr.24373 172.21.15.28:0/3091488821' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:27:12.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:12 smithi066 ceph-mon[25664]: from='mgr.24373 ' entity='mgr.y' 2024-03-21T21:27:13.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:27:13.531 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:27:13.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:13.531 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:13.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:13.531 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:13.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 1 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:27:13.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:27:13.898 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:27:13.898 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:27:13.898 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:27:13.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:27:13.898 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:27:13.898 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-03-21T21:27:13.898 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 1 2024-03-21T21:27:13.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:27:13.898 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:27:14.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:13 smithi028 ceph-mon[29568]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:14.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:13 smithi028 ceph-mon[24806]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:14.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:13 smithi066 ceph-mon[25664]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:14.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph -s' 2024-03-21T21:27:14.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:14 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1170966524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:27:14.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:14 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1170966524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:27:15.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:14 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1170966524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:27:16.122 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:15 smithi028 ceph-mon[29568]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:16.122 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:15 smithi028 ceph-mon[24806]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:16.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:15 smithi066 ceph-mon[25664]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:16.607 INFO:teuthology.orchestra.run.smithi028.stdout: cluster: 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: id: c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: health: HEALTH_OK 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: services: 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: mon: 3 daemons, quorum a,c,b (age 12m) 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: mgr: y(active, since 8m), standbys: x 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 9m) 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: data: 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: pools: 6 pools, 161 pgs 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: objects: 209 objects, 585 KiB 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: usage: 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:16.608 INFO:teuthology.orchestra.run.smithi028.stdout: pgs: 161 active+clean 2024-03-21T21:27:16.609 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:27:16.609 INFO:teuthology.orchestra.run.smithi028.stdout: io: 2024-03-21T21:27:16.609 INFO:teuthology.orchestra.run.smithi028.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-21T21:27:16.609 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:27:16.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:16 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3076145716' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:27:16.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:16 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3076145716' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:27:17.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:27:17.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:16 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3076145716' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:27:18.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:17 smithi028 ceph-mon[24806]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:18.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:17 smithi028 ceph-mon[29568]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:18.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:17 smithi066 ceph-mon[25664]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:19.412 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-03-21T21:27:19.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:19 smithi028 ceph-mon[24806]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:19.413 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:19 smithi028 ceph-mon[29568]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:19.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:19 smithi066 ceph-mon[25664]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:20.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-03-21T21:27:20.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:20 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/477628551' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:27:20.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:20 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/477628551' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:27:20.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:20 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/477628551' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:27:21.402 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:21 smithi028 ceph-mon[29568]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:21.403 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:21 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:21.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:21 smithi028 ceph-mon[24806]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:21.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:21 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:21.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:21 smithi066 ceph-mon[25664]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:21.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:21 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:22.169 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:27:22.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:22 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1217118745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:27:22.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:22 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1217118745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:27:22.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:22 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1217118745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:27:22.784 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:22] "GET /metrics HTTP/1.1" 200 207967 "" "Prometheus/2.33.4" 2024-03-21T21:27:22.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph mgr fail' 2024-03-21T21:27:23.247 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:27:23.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:23 smithi028 ceph-mon[29568]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:23.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:23 smithi028 ceph-mon[24806]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:23.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:27:23.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:23.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:23.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:23.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:23.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:27:23.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:23 smithi066 ceph-mon[25664]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:25.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-mon[24806]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:25.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1092631557' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:27:25.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-mon[24806]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T21:27:25.660 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ignoring --setuser ceph since I am not root 2024-03-21T21:27:25.660 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ignoring --setgroup ceph since I am not root 2024-03-21T21:27:25.661 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-mgr[25042]: -- 172.21.15.28:0/3968192141 <== mon.1 v2:172.21.15.28:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55de80830340 con 0x55de80809000 2024-03-21T21:27:25.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-mon[29568]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:25.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1092631557' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:27:25.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-mon[29568]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T21:27:25.671 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:25 smithi066 ceph-mon[25664]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:25.671 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:25 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1092631557' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:27:25.671 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:25 smithi066 ceph-mon[25664]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T21:27:25.671 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:27:25] ENGINE Bus STOPPING 2024-03-21T21:27:25.671 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:27:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:27:25.672 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:27:25] ENGINE Bus STOPPED 2024-03-21T21:27:25.672 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:27:25] ENGINE Bus STARTING 2024-03-21T21:27:25.672 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:27:25] ENGINE Serving on http://:::9283 2024-03-21T21:27:25.672 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:27:25] ENGINE Bus STARTED 2024-03-21T21:27:26.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'sleep 180' 2024-03-21T21:27:26.193 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:25.948Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-21T21:27:26.194 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:25 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:25.937+0000 7f65268ec000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:27:26.194 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:26.032+0000 7f65268ec000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:27:26.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:27:26.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1092631557' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: mgrmap e22: x(active, starting, since 0.608038s) 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:27:26.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: Manager daemon x is now available 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: Queued rgw.foo for migration 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: Deploying cephadm binary to smithi028 2024-03-21T21:27:26.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[24806]: Deploying cephadm binary to smithi066 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1092631557' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: mgrmap e22: x(active, starting, since 0.608038s) 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:27:26.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: Manager daemon x is now available 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:26.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: Queued rgw.foo for migration 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: Deploying cephadm binary to smithi028 2024-03-21T21:27:26.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-mon[29568]: Deploying cephadm binary to smithi066 2024-03-21T21:27:26.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1092631557' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: mgrmap e22: x(active, starting, since 0.608038s) 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:27:26.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: Manager daemon x is now available 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: Queued rgw.foo for migration 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:27:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:27:26.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:27:26.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:27:26.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: Deploying cephadm binary to smithi028 2024-03-21T21:27:26.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:26 smithi066 ceph-mon[25664]: Deploying cephadm binary to smithi066 2024-03-21T21:27:26.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:26.668+0000 7f65268ec000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:27:26.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:26.760+0000 7f65268ec000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:27:26.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:26 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:26.849+0000 7f65268ec000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:27:27.270 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:27.003+0000 7f65268ec000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:27:27.270 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:27.086+0000 7f65268ec000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:27:27.270 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:27.199+0000 7f65268ec000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:27:27.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-mon[24806]: mgrmap e23: x(active, since 1.62234s) 2024-03-21T21:27:27.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-mon[24806]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:27.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:27.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:27.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:27.287+0000 7f65268ec000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:27:27.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:27.457+0000 7f65268ec000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:27:27.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-mon[29568]: mgrmap e23: x(active, since 1.62234s) 2024-03-21T21:27:27.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-mon[29568]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:27.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:27.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:27.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:27 smithi066 ceph-mon[25664]: mgrmap e23: x(active, since 1.62234s) 2024-03-21T21:27:27.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:27 smithi066 ceph-mon[25664]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:27.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:27 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:27.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:27 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:27.879 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:27 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:27.546+0000 7f65268ec000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:27:28.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:28 smithi028 ceph-mon[24806]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:28.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:28 smithi028 ceph-mon[29568]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:28.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:28 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:28.613+0000 7f65268ec000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:27:28.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:28 smithi066 ceph-mon[25664]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:29.038 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:28 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:28.748+0000 7f65268ec000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:27:29.039 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:28 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:28.946+0000 7f65268ec000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:27:29.302 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:29.039+0000 7f65268ec000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:27:29.303 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:29.186+0000 7f65268ec000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:27:29.693 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:29.523+0000 7f65268ec000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:27:29.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[29568]: mgrmap e24: x(active, since 3s) 2024-03-21T21:27:29.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[29568]: [21/Mar/2024:21:27:28] ENGINE Bus STARTING 2024-03-21T21:27:29.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[29568]: [21/Mar/2024:21:27:28] ENGINE Serving on https://172.21.15.66:7150 2024-03-21T21:27:29.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[29568]: [21/Mar/2024:21:27:28] ENGINE Serving on http://172.21.15.66:8765 2024-03-21T21:27:29.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[29568]: [21/Mar/2024:21:27:28] ENGINE Bus STARTED 2024-03-21T21:27:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[24806]: mgrmap e24: x(active, since 3s) 2024-03-21T21:27:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[24806]: [21/Mar/2024:21:27:28] ENGINE Bus STARTING 2024-03-21T21:27:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[24806]: [21/Mar/2024:21:27:28] ENGINE Serving on https://172.21.15.66:7150 2024-03-21T21:27:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[24806]: [21/Mar/2024:21:27:28] ENGINE Serving on http://172.21.15.66:8765 2024-03-21T21:27:29.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-mon[24806]: [21/Mar/2024:21:27:28] ENGINE Bus STARTED 2024-03-21T21:27:29.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:29 smithi066 ceph-mon[25664]: mgrmap e24: x(active, since 3s) 2024-03-21T21:27:29.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:29 smithi066 ceph-mon[25664]: [21/Mar/2024:21:27:28] ENGINE Bus STARTING 2024-03-21T21:27:29.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:29 smithi066 ceph-mon[25664]: [21/Mar/2024:21:27:28] ENGINE Serving on https://172.21.15.66:7150 2024-03-21T21:27:29.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:29 smithi066 ceph-mon[25664]: [21/Mar/2024:21:27:28] ENGINE Serving on http://172.21.15.66:8765 2024-03-21T21:27:29.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:29 smithi066 ceph-mon[25664]: [21/Mar/2024:21:27:28] ENGINE Bus STARTED 2024-03-21T21:27:30.023 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:29.755+0000 7f65268ec000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:27:30.024 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:29 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:29.845+0000 7f65268ec000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:27:30.425 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:30.025+0000 7f65268ec000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:27:30.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:30 smithi028 ceph-mon[24806]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:30.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:30 smithi028 ceph-mon[29568]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:30.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:30 smithi066 ceph-mon[25664]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:31.313 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:30.987+0000 7f65268ec000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:27:31.313 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:31.314+0000 7f65268ec000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:27:31.582 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:31.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:31.582 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:31.422+0000 7f65268ec000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:27:31.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:31 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:31.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:31.582+0000 7f65268ec000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:27:31.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:27:31] ENGINE Bus STARTING 2024-03-21T21:27:31.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: CherryPy Checker: 2024-03-21T21:27:31.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: The Application mounted at '' has an empty config. 2024-03-21T21:27:31.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: 2024-03-21T21:27:31.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:27:31] ENGINE Serving on http://:::9283 2024-03-21T21:27:31.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: [21/Mar/2024:21:27:31] ENGINE Bus STARTED 2024-03-21T21:27:32.693 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:27:32.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[24806]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:32.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[24806]: Standby manager daemon y started 2024-03-21T21:27:32.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T21:27:32.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:27:32.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T21:27:32.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:27:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[29568]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[29568]: Standby manager daemon y started 2024-03-21T21:27:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T21:27:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:27:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T21:27:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:27:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:32 smithi066 ceph-mon[25664]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:27:32.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:32 smithi066 ceph-mon[25664]: Standby manager daemon y started 2024-03-21T21:27:32.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:32 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T21:27:32.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:32 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:27:32.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:32 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T21:27:32.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:32 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/402140245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:27:33.194 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:32.909Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:33.203 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:32] "GET /metrics HTTP/1.1" 200 34890 "" "Prometheus/2.33.4" 2024-03-21T21:27:33.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:33 smithi066 ceph-mon[25664]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-21T21:27:33.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:33 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:27:33.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:33 smithi028 ceph-mon[29568]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-21T21:27:33.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:33 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:27:33.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:33 smithi028 ceph-mon[24806]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-21T21:27:33.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:33 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:27:33.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:27:33.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:33.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:33.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:27:34.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:34 smithi028 ceph-mon[24806]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:27:34.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:34 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:34.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:34 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:34.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:34 smithi028 ceph-mon[29568]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:27:34.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:34 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:34.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:34 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:34.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:34 smithi066 ceph-mon[25664]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:27:34.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:34 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:34.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:34 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi028 to 2993M 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi028 to 2993M 2024-03-21T21:27:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi028 to 2993M 2024-03-21T21:27:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: Adjusting osd_memory_target on smithi066 to 3248M 2024-03-21T21:27:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: Adjusting osd_memory_target on smithi066 to 3248M 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:27:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:27:37.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:37 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: Adjusting osd_memory_target on smithi066 to 3248M 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:27:37.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:27:37.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:27:37.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:27:37.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:37.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:37 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.599 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:38 smithi028 systemd[1]: Stopping Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:27:38.600 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:38 smithi028 bash[58808]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter.a" found: no such container 2024-03-21T21:27:38.600 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:38 smithi028 podman[58821]: 2024-03-21 21:27:38.409511156 +0000 UTC m=+0.135034252 container died f01f375e6b149e9f9d18262547f2b748ff6c71db0ce33c391bff25e6e9fb23db (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:38.943 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:38 smithi028 podman[58821]: 2024-03-21 21:27:38.756812025 +0000 UTC m=+0.482335119 container cleanup f01f375e6b149e9f9d18262547f2b748ff6c71db0ce33c391bff25e6e9fb23db (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:38.944 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:38 smithi028 bash[58821]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a 2024-03-21T21:27:38.944 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:38 smithi028 bash[58847]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter.a" found: no such container 2024-03-21T21:27:38.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:27:38.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:27:38.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:27:38.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:27:38.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:27:38.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[29568]: Reconfiguring daemon node-exporter.a on smithi028 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:27:38.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:27:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-21T21:27:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:38 smithi028 ceph-mon[24806]: Reconfiguring daemon node-exporter.a on smithi028 2024-03-21T21:27:38.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:27:38.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-21T21:27:38.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:38 smithi066 ceph-mon[25664]: Reconfiguring daemon node-exporter.a on smithi028 2024-03-21T21:27:39.323 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 podman[58835]: 2024-03-21 21:27:39.066587975 +0000 UTC m=+0.660964541 container remove f01f375e6b149e9f9d18262547f2b748ff6c71db0ce33c391bff25e6e9fb23db (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:39.323 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:27:39.323 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-21T21:27:39.323 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 systemd[1]: Stopped Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:27:39.323 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 3.725s CPU time. 2024-03-21T21:27:39.324 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 systemd[1]: Starting Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:27:39.324 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 podman[58917]: 2024-03-21 21:27:39.202658275 +0000 UTC m=+0.017202467 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-21T21:27:39.324 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 podman[58917]: 2024-03-21 21:27:39.325815586 +0000 UTC m=+0.140359770 container create a8a424c65ebefab9d2f10fa68a210fadd3506b4518f36d50f2a34c1f041aea23 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:39.593 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 podman[58917]: 2024-03-21 21:27:39.536785379 +0000 UTC m=+0.351329563 container init a8a424c65ebefab9d2f10fa68a210fadd3506b4518f36d50f2a34c1f041aea23 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:39.593 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 podman[58917]: 2024-03-21 21:27:39.539568612 +0000 UTC m=+0.354112796 container start a8a424c65ebefab9d2f10fa68a210fadd3506b4518f36d50f2a34c1f041aea23 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:39.593 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.543Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-21T21:27:39.593 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.543Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-21T21:27:39.593 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.543Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-21T21:27:39.593 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.543Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T21:27:39.593 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=arp 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-21T21:27:39.594 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=edac 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-21T21:27:39.595 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=os 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-21T21:27:39.596 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=stat 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=time 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=uname 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[58931]: ts=2024-03-21T21:27:39.544Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-21T21:27:39.597 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 bash[58917]: a8a424c65ebefab9d2f10fa68a210fadd3506b4518f36d50f2a34c1f041aea23 2024-03-21T21:27:39.943 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:27:39 smithi028 systemd[1]: Started Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:27:40.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[24806]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:27:40.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:40.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:40.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[24806]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T21:27:40.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[24806]: Reconfiguring daemon alertmanager.a on smithi028 2024-03-21T21:27:40.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:27:40.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[29568]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:27:40.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:40.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:40.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[29568]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T21:27:40.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[29568]: Reconfiguring daemon alertmanager.a on smithi028 2024-03-21T21:27:40.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:40 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:27:40.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:40 smithi066 ceph-mon[25664]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:27:40.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:40 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:40.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:40 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:40.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:40 smithi066 ceph-mon[25664]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T21:27:40.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:40 smithi066 ceph-mon[25664]: Reconfiguring daemon alertmanager.a on smithi028 2024-03-21T21:27:40.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:40 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:27:41.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:41 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:41.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:41 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:41.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:41 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:42.900 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:42 smithi066 ceph-mon[25664]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:27:42.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:42 smithi028 ceph-mon[29568]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:27:42.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:42 smithi028 ceph-mon[24806]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:27:42.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:27:43.248 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:42] "GET /metrics HTTP/1.1" 200 37883 "" "Prometheus/2.33.4" 2024-03-21T21:27:43.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=error ts=2024-03-21T21:27:43.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:43.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:43.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.28:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.28 because it doesn't contain any IP SANs" 2024-03-21T21:27:43.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=warn ts=2024-03-21T21:27:43.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.66:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.66 because it doesn't contain any IP SANs" 2024-03-21T21:27:44.562 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:44 smithi028 ceph-mon[29568]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:27:44.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:44 smithi028 ceph-mon[24806]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:27:44.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:44 smithi066 ceph-mon[25664]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:27:46.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:46 smithi028 ceph-mon[24806]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:46.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:46 smithi028 ceph-mon[29568]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:46.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:46 smithi066 ceph-mon[25664]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:47.326 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:47 smithi028 systemd[1]: Stopping Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:27:47.326 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:47 smithi028 bash[59181]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-21T21:27:47.617 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[49593]: level=info ts=2024-03-21T21:27:47.326Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:27:47.617 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:47 smithi028 podman[59194]: 2024-03-21 21:27:47.347286293 +0000 UTC m=+0.130146914 container died 6d5934565000b0297e8a8e9203ef0bb72be71bf8a3bfae2d30801be2207322f3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:47.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:47 smithi028 podman[59194]: 2024-03-21 21:27:47.727349639 +0000 UTC m=+0.510210259 container cleanup 6d5934565000b0297e8a8e9203ef0bb72be71bf8a3bfae2d30801be2207322f3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:47.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:47 smithi028 bash[59194]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T21:27:47.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:47 smithi028 bash[59221]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-21T21:27:48.294 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:47 smithi028 podman[59208]: 2024-03-21 21:27:47.986620369 +0000 UTC m=+0.638398828 container remove 6d5934565000b0297e8a8e9203ef0bb72be71bf8a3bfae2d30801be2207322f3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:48.294 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 podman[59208]: 2024-03-21 21:27:48.062514936 +0000 UTC m=+0.714293396 volume remove e5cf90f4bd4d85f568e0426fd6abde58ea8d12e07df6ab528711dba0b7672859 2024-03-21T21:27:48.294 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T21:27:48.294 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 systemd[1]: Stopped Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:27:48.295 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.046s CPU time. 2024-03-21T21:27:48.295 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 systemd[1]: Starting Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:27:48.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-mon[29568]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:48.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-mon[24806]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:48.555 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 podman[59295]: 2024-03-21 21:27:48.296758196 +0000 UTC m=+0.105450219 volume create 2cbb7777266b2508439c73a6b713a217a836ddb2c635b4cbd93b43bbf6323e3f 2024-03-21T21:27:48.555 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 podman[59295]: 2024-03-21 21:27:48.209638197 +0000 UTC m=+0.018330177 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-21T21:27:48.555 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 podman[59295]: 2024-03-21 21:27:48.388891168 +0000 UTC m=+0.197583147 container create cd065b97fad114063b61fc577e63d81c1e3d33853f5b83f2649fac9c95527555 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:48.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:48 smithi066 ceph-mon[25664]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:48.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 podman[59295]: 2024-03-21 21:27:48.649916798 +0000 UTC m=+0.458608788 container init cd065b97fad114063b61fc577e63d81c1e3d33853f5b83f2649fac9c95527555 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:48.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 podman[59295]: 2024-03-21 21:27:48.652621514 +0000 UTC m=+0.461313499 container start cd065b97fad114063b61fc577e63d81c1e3d33853f5b83f2649fac9c95527555 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:27:48.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:48.664Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-21T21:27:48.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:48.664Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-21T21:27:48.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:48.664Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.28 port=9094 2024-03-21T21:27:48.945 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:48.666Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T21:27:48.945 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:48.693Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T21:27:48.945 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:48.693Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T21:27:48.945 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:48.696Z caller=main.go:518 msg=Listening address=:9093 2024-03-21T21:27:48.945 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:48.696Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-21T21:27:48.945 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 bash[59295]: cd065b97fad114063b61fc577e63d81c1e3d33853f5b83f2649fac9c95527555 2024-03-21T21:27:48.945 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:48 smithi028 systemd[1]: Started Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:27:49.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:49.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:27:49.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:27:49.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:27:49.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:49 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:27:50.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:49 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:50.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:49 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:50.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:49 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:27:50.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:49 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:27:50.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:49 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:27:50.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:49 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:27:50.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:49 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:27:50.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:50 smithi028 ceph-mon[24806]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:50.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:50 smithi028 ceph-mon[29568]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:50.944 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:50.667Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000733289s 2024-03-21T21:27:51.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:50 smithi066 ceph-mon[25664]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:52.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:51 smithi028 ceph-mon[24806]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:52.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:51 smithi028 ceph-mon[29568]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:52.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:51 smithi066 ceph-mon[25664]: from='client.14841 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:27:52.747 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:27:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:27:52.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.66:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.28:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:27:52.774 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:27:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:27:53.047 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:52 smithi028 ceph-mon[29568]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:53.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:52 smithi028 ceph-mon[24806]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:53.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:52 smithi066 ceph-mon[25664]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:27:53.248 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:27:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:27:52] "GET /metrics HTTP/1.1" 200 37883 "" "Prometheus/2.33.4" 2024-03-21T21:27:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[29568]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:27:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[29568]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[29568]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:27:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3917364379' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:27:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/676579653' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1624152288"}]: dispatch 2024-03-21T21:27:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1624152288"}]: dispatch 2024-03-21T21:27:54.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:54.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:54.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[24806]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:27:54.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[24806]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:54.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[24806]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:27:54.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3917364379' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:27:54.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/676579653' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1624152288"}]: dispatch 2024-03-21T21:27:54.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:54 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1624152288"}]: dispatch 2024-03-21T21:27:54.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:54 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:54.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:54 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:27:54.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:54 smithi066 ceph-mon[25664]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:27:54.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:54 smithi066 ceph-mon[25664]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:27:54.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:54 smithi066 ceph-mon[25664]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:27:54.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:54 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3917364379' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:27:54.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:54 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/676579653' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1624152288"}]: dispatch 2024-03-21T21:27:54.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:54 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1624152288"}]: dispatch 2024-03-21T21:27:55.680 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:55 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1624152288"}]': finished 2024-03-21T21:27:55.680 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:55 smithi066 ceph-mon[25664]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T21:27:55.680 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:55 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2754117777' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1698649293"}]: dispatch 2024-03-21T21:27:55.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:55 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1624152288"}]': finished 2024-03-21T21:27:55.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:55 smithi028 ceph-mon[24806]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T21:27:55.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:55 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2754117777' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1698649293"}]: dispatch 2024-03-21T21:27:55.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:55 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1624152288"}]': finished 2024-03-21T21:27:55.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:55 smithi028 ceph-mon[29568]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T21:27:55.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:55 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2754117777' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1698649293"}]: dispatch 2024-03-21T21:27:56.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2754117777' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1698649293"}]': finished 2024-03-21T21:27:56.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[29568]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T21:27:56.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[29568]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:27:56.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:27:56.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1886656163' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1317986469"}]: dispatch 2024-03-21T21:27:56.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1317986469"}]: dispatch 2024-03-21T21:27:56.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2754117777' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1698649293"}]': finished 2024-03-21T21:27:56.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[24806]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T21:27:56.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[24806]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:27:56.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:27:56.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1886656163' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1317986469"}]: dispatch 2024-03-21T21:27:56.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:56 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1317986469"}]: dispatch 2024-03-21T21:27:56.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:56 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2754117777' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1698649293"}]': finished 2024-03-21T21:27:56.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:56 smithi066 ceph-mon[25664]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T21:27:56.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:56 smithi066 ceph-mon[25664]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:27:56.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:56 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:27:56.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:56 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1886656163' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1317986469"}]: dispatch 2024-03-21T21:27:56.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:56 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1317986469"}]: dispatch 2024-03-21T21:27:57.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:57 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1317986469"}]': finished 2024-03-21T21:27:57.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:57 smithi028 ceph-mon[24806]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T21:27:57.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:57 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1826586487' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/3101059707"}]: dispatch 2024-03-21T21:27:57.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:57 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/3101059707"}]: dispatch 2024-03-21T21:27:57.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:57 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1317986469"}]': finished 2024-03-21T21:27:57.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:57 smithi028 ceph-mon[29568]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T21:27:57.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:57 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1826586487' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/3101059707"}]: dispatch 2024-03-21T21:27:57.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:57 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/3101059707"}]: dispatch 2024-03-21T21:27:57.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:57 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1317986469"}]': finished 2024-03-21T21:27:57.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:57 smithi066 ceph-mon[25664]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T21:27:57.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:57 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1826586487' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/3101059707"}]: dispatch 2024-03-21T21:27:57.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:57 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/3101059707"}]: dispatch 2024-03-21T21:27:58.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/3101059707"}]': finished 2024-03-21T21:27:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[24806]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T21:27:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[24806]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T21:27:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3477283815' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/3101059707"}]: dispatch 2024-03-21T21:27:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/3101059707"}]: dispatch 2024-03-21T21:27:58.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/3101059707"}]': finished 2024-03-21T21:27:58.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[29568]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T21:27:58.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[29568]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T21:27:58.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3477283815' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/3101059707"}]: dispatch 2024-03-21T21:27:58.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/3101059707"}]: dispatch 2024-03-21T21:27:58.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:58 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/3101059707"}]': finished 2024-03-21T21:27:58.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:58 smithi066 ceph-mon[25664]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T21:27:58.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:58 smithi066 ceph-mon[25664]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T21:27:58.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:58 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3477283815' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/3101059707"}]: dispatch 2024-03-21T21:27:58.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:58 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/3101059707"}]: dispatch 2024-03-21T21:27:58.943 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:27:58 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:27:58.669Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00277655s 2024-03-21T21:27:59.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:59 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/3101059707"}]': finished 2024-03-21T21:27:59.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:59 smithi028 ceph-mon[24806]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T21:27:59.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:27:59 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/994174034' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/4172892718"}]: dispatch 2024-03-21T21:27:59.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:59 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/3101059707"}]': finished 2024-03-21T21:27:59.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:59 smithi028 ceph-mon[29568]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T21:27:59.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:27:59 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/994174034' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/4172892718"}]: dispatch 2024-03-21T21:27:59.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:59 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/3101059707"}]': finished 2024-03-21T21:27:59.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:59 smithi066 ceph-mon[25664]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T21:27:59.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:27:59 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/994174034' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/4172892718"}]: dispatch 2024-03-21T21:28:00.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:00 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/994174034' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/4172892718"}]': finished 2024-03-21T21:28:00.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:00 smithi028 ceph-mon[24806]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T21:28:00.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:00 smithi028 ceph-mon[24806]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T21:28:00.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:00 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2581044354' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1106126851"}]: dispatch 2024-03-21T21:28:00.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:00 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/994174034' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/4172892718"}]': finished 2024-03-21T21:28:00.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:00 smithi028 ceph-mon[29568]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T21:28:00.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:00 smithi028 ceph-mon[29568]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T21:28:00.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:00 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2581044354' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1106126851"}]: dispatch 2024-03-21T21:28:00.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:00 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/994174034' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/4172892718"}]': finished 2024-03-21T21:28:00.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:00 smithi066 ceph-mon[25664]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T21:28:00.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:00 smithi066 ceph-mon[25664]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T21:28:00.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:00 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2581044354' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1106126851"}]: dispatch 2024-03-21T21:28:01.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:01 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2581044354' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1106126851"}]': finished 2024-03-21T21:28:01.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:01 smithi066 ceph-mon[25664]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T21:28:01.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:01 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2581044354' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1106126851"}]': finished 2024-03-21T21:28:01.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:01 smithi028 ceph-mon[29568]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T21:28:01.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:01 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2581044354' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1106126851"}]': finished 2024-03-21T21:28:01.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:01 smithi028 ceph-mon[24806]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T21:28:02.630 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:02 smithi066 ceph-mon[25664]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:28:02.630 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:02.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.66:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.28:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:02.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:02 smithi028 ceph-mon[29568]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:28:02.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:02 smithi028 ceph-mon[24806]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 64 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:28:02.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:28:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[25037]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:28:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:28:03.190 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:28:02] "GET /metrics HTTP/1.1" 200 37883 "" "Prometheus/2.33.4" 2024-03-21T21:28:04.248 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 systemd[1]: Stopping Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:28:04.248 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 bash[49129]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:28:04.248 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.170Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:28:04.248 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.170Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T21:28:04.248 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.170Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.170Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.170Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.170Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.170Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.170Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.170Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.171Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.171Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[42603]: ts=2024-03-21T21:28:04.171Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T21:28:04.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 podman[49141]: 2024-03-21 21:28:04.180605347 +0000 UTC m=+0.117901050 container died c61488efcefc5e06d2403f1feec31090162c1f1b72456c7c79d549a5162ae62c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:28:04.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:04 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:04.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:04 smithi028 ceph-mon[29568]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:04.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:04 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:04.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:04 smithi028 ceph-mon[24806]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:04.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:04.735 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:04 smithi066 ceph-mon[25664]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:04.735 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 podman[49141]: 2024-03-21 21:28:04.511473353 +0000 UTC m=+0.448769064 container cleanup c61488efcefc5e06d2403f1feec31090162c1f1b72456c7c79d549a5162ae62c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:28:04.735 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 bash[49141]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T21:28:04.993 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 podman[49155]: 2024-03-21 21:28:04.737551013 +0000 UTC m=+0.554035433 container remove c61488efcefc5e06d2403f1feec31090162c1f1b72456c7c79d549a5162ae62c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:28:04.993 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 bash[49169]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:28:04.993 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T21:28:04.993 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 systemd[1]: Stopped Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:04.993 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 2.075s CPU time. 2024-03-21T21:28:04.993 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 systemd[1]: Starting Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:28:05.247 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:04 smithi066 podman[49233]: 2024-03-21 21:28:04.899800617 +0000 UTC m=+0.020101935 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T21:28:05.248 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 podman[49233]: 2024-03-21 21:28:05.022114288 +0000 UTC m=+0.142415593 container create cc8b8c4750568d403064206ce42731b0b2020bb9b0d7e2767afb16b1ccfde2bc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:28:05.248 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 podman[49233]: 2024-03-21 21:28:05.240796669 +0000 UTC m=+0.361097987 container init cc8b8c4750568d403064206ce42731b0b2020bb9b0d7e2767afb16b1ccfde2bc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:28:05.248 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 podman[49233]: 2024-03-21 21:28:05.243367847 +0000 UTC m=+0.363669149 container start cc8b8c4750568d403064206ce42731b0b2020bb9b0d7e2767afb16b1ccfde2bc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:28:05.739 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.267Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T21:28:05.739 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.267Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T21:28:05.739 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.267Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T21:28:05.739 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.267Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi066 (none))" 2024-03-21T21:28:05.739 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.267Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T21:28:05.739 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.267Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.269Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.269Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.271Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.273Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.273Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.667µs 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.273Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 bash[49233]: cc8b8c4750568d403064206ce42731b0b2020bb9b0d7e2767afb16b1ccfde2bc 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 systemd[1]: Started Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.478Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.741Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-03-21T21:28:05.740 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.741Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-03-21T21:28:05.997 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:05 smithi066 systemd[1]: Stopping Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:28:05.997 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:05 smithi066 bash[49360]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter.b" found: no such container 2024-03-21T21:28:05.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.741Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.703µs wal_replay_duration=468.320289ms total_replay_duration=468.367601ms 2024-03-21T21:28:05.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.746Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T21:28:05.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.746Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T21:28:05.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.746Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T21:28:05.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.773Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=26.621095ms db_storage=862ns remote_storage=1.767µs web_handler=682ns query_engine=734ns scrape=520.987µs scrape_sd=155.007µs notify=18.317µs notify_sd=21.689µs rules=25.366884ms 2024-03-21T21:28:05.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:05.773Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T21:28:06.371 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 podman[49373]: 2024-03-21 21:28:06.130070209 +0000 UTC m=+0.149385468 container died c03593861fb95ff868b5231ecca09418e66333244131586826d3981760d50c69 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:28:06.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:06.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:06.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[24806]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-21T21:28:06.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[24806]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-21T21:28:06.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[24806]: Reconfiguring daemon node-exporter.b on smithi066 2024-03-21T21:28:06.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:06.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:06.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[29568]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-21T21:28:06.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[29568]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-21T21:28:06.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:06 smithi028 ceph-mon[29568]: Reconfiguring daemon node-exporter.b on smithi066 2024-03-21T21:28:06.747 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 podman[49373]: 2024-03-21 21:28:06.469458403 +0000 UTC m=+0.488773662 container cleanup c03593861fb95ff868b5231ecca09418e66333244131586826d3981760d50c69 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:28:06.747 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 bash[49373]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b 2024-03-21T21:28:06.748 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 bash[49402]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter.b" found: no such container 2024-03-21T21:28:06.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:06 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:06.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:06 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:06.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:06 smithi066 ceph-mon[25664]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-21T21:28:06.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:06 smithi066 ceph-mon[25664]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-21T21:28:06.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:06 smithi066 ceph-mon[25664]: Reconfiguring daemon node-exporter.b on smithi066 2024-03-21T21:28:07.095 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 podman[49387]: 2024-03-21 21:28:06.770539877 +0000 UTC m=+0.644842339 container remove c03593861fb95ff868b5231ecca09418e66333244131586826d3981760d50c69 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:28:07.095 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:28:07.095 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-21T21:28:07.095 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 systemd[1]: Stopped Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:07.095 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 4.054s CPU time. 2024-03-21T21:28:07.096 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 systemd[1]: Starting Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:28:07.096 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:06 smithi066 podman[49474]: 2024-03-21 21:28:06.980100501 +0000 UTC m=+0.095108003 container create f6cdb9cb99234cd66daaffac9ebaf9ded30d1bd67908b7145b4c9f6736a9dcaf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:28:07.096 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 podman[49474]: 2024-03-21 21:28:06.903890807 +0000 UTC m=+0.018898309 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 podman[49474]: 2024-03-21 21:28:07.182466658 +0000 UTC m=+0.297474169 container init f6cdb9cb99234cd66daaffac9ebaf9ded30d1bd67908b7145b4c9f6736a9dcaf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 podman[49474]: 2024-03-21 21:28:07.185072264 +0000 UTC m=+0.300079769 container start f6cdb9cb99234cd66daaffac9ebaf9ded30d1bd67908b7145b4c9f6736a9dcaf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.188Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.188Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.188Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.188Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=arp 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-21T21:28:07.498 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-21T21:28:07.499 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-21T21:28:07.499 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-21T21:28:07.499 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-21T21:28:07.499 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-21T21:28:07.499 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-21T21:28:07.499 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-21T21:28:07.499 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=edac 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-21T21:28:07.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=os 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=stat 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-21T21:28:07.501 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=time 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=uname 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[49489]: ts=2024-03-21T21:28:07.189Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 bash[49474]: f6cdb9cb99234cd66daaffac9ebaf9ded30d1bd67908b7145b4c9f6736a9dcaf 2024-03-21T21:28:07.502 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:28:07 smithi066 systemd[1]: Started Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:08.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: Reconfiguring grafana.a (dependencies changed)... 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: Reconfiguring daemon grafana.a on smithi066 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[29568]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:08.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: Reconfiguring grafana.a (dependencies changed)... 2024-03-21T21:28:08.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T21:28:08.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:28:08.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:28:08.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: Reconfiguring daemon grafana.a on smithi066 2024-03-21T21:28:08.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:08 smithi028 ceph-mon[24806]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:08.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: Reconfiguring grafana.a (dependencies changed)... 2024-03-21T21:28:08.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T21:28:08.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:08.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:28:08.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T21:28:08.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: Reconfiguring daemon grafana.a on smithi066 2024-03-21T21:28:08.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:08 smithi066 ceph-mon[25664]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:10.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:10 smithi066 ceph-mon[25664]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:28:10.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:10 smithi028 ceph-mon[29568]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:28:10.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:10 smithi028 ceph-mon[24806]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:28:11.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:11 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:11.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:11 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:11.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:11 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:12.745 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:12 smithi066 ceph-mon[25664]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-03-21T21:28:12.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:12 smithi028 ceph-mon[29568]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-03-21T21:28:12.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:12 smithi028 ceph-mon[24806]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-03-21T21:28:12.997 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:28:12] "GET /metrics HTTP/1.1" 200 37881 "" "Prometheus/2.33.4" 2024-03-21T21:28:13.997 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:14.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:14 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:14.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:14 smithi028 ceph-mon[24806]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:14.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:14 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:14.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:14 smithi028 ceph-mon[29568]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:14.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:14 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:14.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:14 smithi066 ceph-mon[25664]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:16.670 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:16 smithi066 ceph-mon[25664]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:16.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:16 smithi028 ceph-mon[29568]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:16.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:16 smithi028 ceph-mon[24806]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:17.998 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:17 smithi066 systemd[1]: Stopping Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:28:17.998 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:17 smithi066 bash[49769]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana.a" found: no such container 2024-03-21T21:28:17.998 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:28:17+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-21T21:28:17.998 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[41799]: t=2024-03-21T21:28:17+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-03-21T21:28:17.998 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:17 smithi066 podman[49780]: 2024-03-21 21:28:17.827437171 +0000 UTC m=+0.140588455 container died a550093f473a589f57cd0a8b419e58abd666aa4950a680e7b10f571b12088c3c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, version=8.5, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, name=ubi8, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc.) 2024-03-21T21:28:18.388 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 podman[49780]: 2024-03-21 21:28:18.181639823 +0000 UTC m=+0.494791107 container cleanup a550093f473a589f57cd0a8b419e58abd666aa4950a680e7b10f571b12088c3c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, architecture=x86_64, io.openshift.tags=base rhel8, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, vendor=Red Hat, Inc., distribution-scope=public, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container) 2024-03-21T21:28:18.389 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 bash[49780]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a 2024-03-21T21:28:18.389 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 bash[49809]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana.a" found: no such container 2024-03-21T21:28:18.639 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:18 smithi066 ceph-mon[25664]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:18.639 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 podman[49794]: 2024-03-21 21:28:18.390937764 +0000 UTC m=+0.560797023 container remove a550093f473a589f57cd0a8b419e58abd666aa4950a680e7b10f571b12088c3c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , io.buildah.version=1.24.2, vcs-type=git, description=Ceph Grafana Container, name=ubi8, version=8.5, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-21T21:28:18.640 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-21T21:28:18.640 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 systemd[1]: Stopped Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:18.640 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Consumed 3.453s CPU time. 2024-03-21T21:28:18.640 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 systemd[1]: Starting Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:28:18.640 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 podman[49875]: 2024-03-21 21:28:18.53596317 +0000 UTC m=+0.018300276 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T21:28:18.640 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 podman[49875]: 2024-03-21 21:28:18.641884583 +0000 UTC m=+0.124221723 container create 88e6f16f9c3c4eecd3a281d433828a8413d973e3f6df2ba93a75cf7fb95e7684 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, maintainer=Paul Cuzner , vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, io.buildah.version=1.24.2, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-21T21:28:18.915 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:18.915 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 podman[49875]: 2024-03-21 21:28:18.886415004 +0000 UTC m=+0.368752103 container init 88e6f16f9c3c4eecd3a281d433828a8413d973e3f6df2ba93a75cf7fb95e7684 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.buildah.version=1.24.2, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , version=8.5, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, name=ubi8, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-21T21:28:18.915 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 podman[49875]: 2024-03-21 21:28:18.88899385 +0000 UTC m=+0.371330952 container start 88e6f16f9c3c4eecd3a281d433828a8413d973e3f6df2ba93a75cf7fb95e7684 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, distribution-scope=public, vendor=Red Hat, Inc., io.openshift.expose-services=, io.buildah.version=1.24.2, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, architecture=x86_64) 2024-03-21T21:28:18.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:18 smithi028 ceph-mon[29568]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:18.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:18 smithi028 ceph-mon[24806]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:19.186 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STOPPING 2024-03-21T21:28:19.186 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:28:19.187 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STOPPED 2024-03-21T21:28:19.187 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STARTING 2024-03-21T21:28:19.187 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Serving on http://:::9283 2024-03-21T21:28:19.187 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STARTED 2024-03-21T21:28:19.187 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[49890]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:28:19.187 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[49890]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:28:19.187 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 bash[49875]: 88e6f16f9c3c4eecd3a281d433828a8413d973e3f6df2ba93a75cf7fb95e7684 2024-03-21T21:28:19.187 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 systemd[1]: Started Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:19.187 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:18 smithi066 podman[49906]: 2024-03-21 21:28:18.996951416 +0000 UTC m=+0.072827880 container died 88e6f16f9c3c4eecd3a281d433828a8413d973e3f6df2ba93a75cf7fb95e7684 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, version=8.5, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, description=Ceph Grafana Container, io.buildah.version=1.24.2, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-type=git, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, vendor=Red Hat, Inc.) 2024-03-21T21:28:19.497 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STOPPING 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STOPPED 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STARTING 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Serving on http://:::9283 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STARTED 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STOPPING 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STOPPED 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STARTING 2024-03-21T21:28:19.498 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Serving on http://:::9283 2024-03-21T21:28:19.499 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:28:19] ENGINE Bus STARTED 2024-03-21T21:28:19.976 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:19 smithi066 podman[49906]: 2024-03-21 21:28:19.565145931 +0000 UTC m=+0.641022393 container remove 88e6f16f9c3c4eecd3a281d433828a8413d973e3f6df2ba93a75cf7fb95e7684 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vendor=Red Hat, Inc., io.buildah.version=1.24.2, vcs-type=git, architecture=x86_64, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, distribution-scope=public, com.redhat.component=ubi8-container, release=236.1648460182, build-date=2022-03-28T10:36:18.413762) 2024-03-21T21:28:19.976 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:19 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T21:28:19.976 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:19 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:28:20.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:28:20.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi066.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T21:28:20.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:28:20.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi066.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T21:28:20.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:20 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:28:20.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:28:20.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi066.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi066.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:28:20.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:28:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi066.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T21:28:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:28:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi066.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T21:28:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:20 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:28:21.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:28:21.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T21:28:21.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:28:21.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: Adding iSCSI gateway http://:@172.21.15.28:5000 to Dashboard 2024-03-21T21:28:21.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:28:21.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:28:21.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:28:21.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi066.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T21:28:21.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi066.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[24806]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: Adding iSCSI gateway http://:@172.21.15.28:5000 to Dashboard 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi066.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:28:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi066.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T21:28:21.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:21 smithi028 ceph-mon[29568]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: Adding iSCSI gateway http://:@172.21.15.28:5000 to Dashboard 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi066.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi066.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T21:28:21.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:21 smithi066 ceph-mon[25664]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:22.997 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:28:22] "GET /metrics HTTP/1.1" 200 37881 "" "Prometheus/2.33.4" 2024-03-21T21:28:23.341 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[29568]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:23.341 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.341 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.341 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[24806]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:28:23.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:28:23.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:23 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:23 smithi066 ceph-mon[25664]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:23.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:23 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:23 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:23 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:23 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:23 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:28:23.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:23 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:28:23.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:23 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:23.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:24.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:24 smithi028 ceph-mon[29568]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:28:24.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:24 smithi028 ceph-mon[29568]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:28:24.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:24 smithi028 ceph-mon[24806]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:28:24.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:24 smithi028 ceph-mon[24806]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:28:24.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:24 smithi066 ceph-mon[25664]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:28:24.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:24 smithi066 ceph-mon[25664]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:28:25.399 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:25 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:25.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:25 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:25.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:25 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:26.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:26 smithi028 ceph-mon[29568]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-21T21:28:26.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:26 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:26.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:26.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:26 smithi028 ceph-mon[24806]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-21T21:28:26.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:26 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:26.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:26.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:26 smithi066 ceph-mon[25664]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-21T21:28:26.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:26 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:28:26.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:28.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:28 smithi028 ceph-mon[29568]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:28:28.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:28 smithi028 ceph-mon[24806]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:28:28.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:28 smithi066 ceph-mon[25664]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:28:28.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:29.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:29 smithi028 ceph-mon[24806]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-21T21:28:29.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:29 smithi028 ceph-mon[29568]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-21T21:28:29.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:29 smithi066 ceph-mon[25664]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-21T21:28:30.038 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:29 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-03-21T21:28:30.038 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:29 smithi066 systemd[1]: Stopped Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:30.038 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:29 smithi066 systemd[1]: Starting Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:28:30.039 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:29 smithi066 podman[50578]: 2024-03-21 21:28:29.89960773 +0000 UTC m=+0.018639371 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T21:28:30.349 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 podman[50578]: 2024-03-21 21:28:30.040597524 +0000 UTC m=+0.159629202 container create 9e6ef20566e76a74214ca0ee9183ff4b2ab7849be164423c842b417332bfdfcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, distribution-scope=public, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, version=8.5, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, maintainer=Paul Cuzner , description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182) 2024-03-21T21:28:30.349 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 podman[50578]: 2024-03-21 21:28:30.251260324 +0000 UTC m=+0.370291961 container init 9e6ef20566e76a74214ca0ee9183ff4b2ab7849be164423c842b417332bfdfcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, name=ubi8, distribution-scope=public, description=Ceph Grafana Container, maintainer=Paul Cuzner , release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.expose-services=, io.buildah.version=1.24.2, vendor=Red Hat, Inc., version=8.5, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-21T21:28:30.349 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 podman[50578]: 2024-03-21 21:28:30.254313232 +0000 UTC m=+0.373344877 container start 9e6ef20566e76a74214ca0ee9183ff4b2ab7849be164423c842b417332bfdfcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, com.redhat.component=ubi8-container, distribution-scope=public, maintainer=Paul Cuzner , io.openshift.tags=base rhel8) 2024-03-21T21:28:30.350 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[50593]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:28:30.350 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[50593]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:28:30.350 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 bash[50578]: 9e6ef20566e76a74214ca0ee9183ff4b2ab7849be164423c842b417332bfdfcc 2024-03-21T21:28:30.748 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 systemd[1]: Started Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:30.748 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 podman[50609]: 2024-03-21 21:28:30.363431346 +0000 UTC m=+0.072952126 container died 9e6ef20566e76a74214ca0ee9183ff4b2ab7849be164423c842b417332bfdfcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, version=8.5, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , release=236.1648460182, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, com.redhat.component=ubi8-container, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-21T21:28:31.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 podman[50609]: 2024-03-21 21:28:30.941471149 +0000 UTC m=+0.650991939 container remove 9e6ef20566e76a74214ca0ee9183ff4b2ab7849be164423c842b417332bfdfcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, architecture=x86_64, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.buildah.version=1.24.2, name=ubi8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, description=Ceph Grafana Container, release=236.1648460182) 2024-03-21T21:28:31.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T21:28:31.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:30 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T21:28:32.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:31 smithi066 ceph-mon[25664]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-21T21:28:32.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:31 smithi028 ceph-mon[24806]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-21T21:28:32.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:31 smithi028 ceph-mon[29568]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-21T21:28:32.997 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:28:32] "GET /metrics HTTP/1.1" 200 37957 "" "Prometheus/2.33.4" 2024-03-21T21:28:33.987 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:34.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:33 smithi066 ceph-mon[25664]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:28:34.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:33 smithi028 ceph-mon[24806]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:28:34.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:33 smithi028 ceph-mon[29568]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:28:35.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:34 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:35.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:34 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:35.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:34 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:36.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:35 smithi066 ceph-mon[25664]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:36.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:35 smithi028 ceph-mon[29568]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:36.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:35 smithi028 ceph-mon[24806]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:38.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:37 smithi066 ceph-mon[25664]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:38.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:37 smithi028 ceph-mon[29568]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:38.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:37 smithi028 ceph-mon[24806]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:38.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:39.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:39 smithi028 ceph-mon[29568]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:39.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:39 smithi028 ceph-mon[24806]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:39.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:39 smithi066 ceph-mon[25664]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:40.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:40 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:40.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:40 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:40.997 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:40 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:41.350 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-03-21T21:28:41.350 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 systemd[1]: Stopped Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:41.350 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 systemd[1]: Starting Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:28:41.351 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 podman[50681]: 2024-03-21 21:28:41.145358519 +0000 UTC m=+0.018150360 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T21:28:41.351 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 podman[50681]: 2024-03-21 21:28:41.248803068 +0000 UTC m=+0.121594891 container create fb6df91caa34b5e25be3f0fc19e9e518174b312402e39096f1f7512cf9ebbd2a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, io.openshift.expose-services=, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., architecture=x86_64, distribution-scope=public, io.buildah.version=1.24.2, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, build-date=2022-03-28T10:36:18.413762, version=8.5) 2024-03-21T21:28:41.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:41 smithi066 ceph-mon[25664]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:41.748 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 podman[50681]: 2024-03-21 21:28:41.459470491 +0000 UTC m=+0.332262324 container init fb6df91caa34b5e25be3f0fc19e9e518174b312402e39096f1f7512cf9ebbd2a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, distribution-scope=public, io.buildah.version=1.24.2, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5) 2024-03-21T21:28:41.748 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 podman[50681]: 2024-03-21 21:28:41.462203252 +0000 UTC m=+0.334995076 container start fb6df91caa34b5e25be3f0fc19e9e518174b312402e39096f1f7512cf9ebbd2a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, architecture=x86_64, distribution-scope=public, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, description=Ceph Grafana Container, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2) 2024-03-21T21:28:41.748 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[50696]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:28:41.748 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[50696]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:28:41.748 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 bash[50681]: fb6df91caa34b5e25be3f0fc19e9e518174b312402e39096f1f7512cf9ebbd2a 2024-03-21T21:28:41.748 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 systemd[1]: Started Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:41.748 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:41 smithi066 podman[50712]: 2024-03-21 21:28:41.547983319 +0000 UTC m=+0.052722851 container died fb6df91caa34b5e25be3f0fc19e9e518174b312402e39096f1f7512cf9ebbd2a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, vcs-type=git, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., distribution-scope=public) 2024-03-21T21:28:41.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:41 smithi028 ceph-mon[24806]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:41.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:41 smithi028 ceph-mon[29568]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:42.498 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:42 smithi066 podman[50712]: 2024-03-21 21:28:42.221435064 +0000 UTC m=+0.726174638 container remove fb6df91caa34b5e25be3f0fc19e9e518174b312402e39096f1f7512cf9ebbd2a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, vendor=Red Hat, Inc., maintainer=Paul Cuzner , vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, io.openshift.expose-services=, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, name=ubi8) 2024-03-21T21:28:42.498 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:42 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T21:28:42.498 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:42 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T21:28:42.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:28:42] "GET /metrics HTTP/1.1" 200 37957 "" "Prometheus/2.33.4" 2024-03-21T21:28:43.989 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:44.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:43 smithi066 ceph-mon[25664]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:44.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:43 smithi028 ceph-mon[24806]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:44.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:43 smithi028 ceph-mon[29568]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:45.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:44 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:44 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:45.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:44 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:46.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:45 smithi066 ceph-mon[25664]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:46.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:45 smithi028 ceph-mon[24806]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:46.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:45 smithi028 ceph-mon[29568]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:48.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:48 smithi028 ceph-mon[24806]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:48.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:48 smithi028 ceph-mon[29568]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:48.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:48 smithi066 ceph-mon[25664]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:48.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:48 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:49.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:49 smithi028 ceph-mon[24806]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:49.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:49 smithi028 ceph-mon[29568]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:49.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:49 smithi066 ceph-mon[25664]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:52.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:51 smithi066 ceph-mon[25664]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:52.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:51 smithi028 ceph-mon[24806]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:52.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:51 smithi028 ceph-mon[29568]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:52.636 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-03-21T21:28:52.636 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 systemd[1]: Stopped Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:52.636 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 systemd[1]: Starting Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:28:52.636 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 podman[50783]: 2024-03-21 21:28:52.394950703 +0000 UTC m=+0.018205043 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T21:28:52.636 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 podman[50783]: 2024-03-21 21:28:52.638532513 +0000 UTC m=+0.261786840 container create a2c368760f2573062c039efb245cccfa316fa87b82712d6acf1fbbf85214a2fe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, vcs-type=git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, maintainer=Paul Cuzner , release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, vendor=Red Hat, Inc.) 2024-03-21T21:28:52.888 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:28:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:28:52] "GET /metrics HTTP/1.1" 200 37957 "" "Prometheus/2.33.4" 2024-03-21T21:28:53.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 podman[50783]: 2024-03-21 21:28:52.891042275 +0000 UTC m=+0.514296616 container init a2c368760f2573062c039efb245cccfa316fa87b82712d6acf1fbbf85214a2fe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, release=236.1648460182, distribution-scope=public, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vendor=Red Hat, Inc., version=8.5, name=ubi8, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, vcs-type=git, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64) 2024-03-21T21:28:53.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 podman[50783]: 2024-03-21 21:28:52.893881855 +0000 UTC m=+0.517136181 container start a2c368760f2573062c039efb245cccfa316fa87b82712d6acf1fbbf85214a2fe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.component=ubi8-container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.openshift.expose-services=, version=8.5, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-21T21:28:53.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[50797]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:28:53.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[50797]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:28:53.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:52 smithi066 bash[50783]: a2c368760f2573062c039efb245cccfa316fa87b82712d6acf1fbbf85214a2fe 2024-03-21T21:28:53.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:53 smithi066 systemd[1]: Started Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:28:53.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:53 smithi066 podman[50813]: 2024-03-21 21:28:53.025454534 +0000 UTC m=+0.098002366 container died a2c368760f2573062c039efb245cccfa316fa87b82712d6acf1fbbf85214a2fe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.openshift.tags=base rhel8, release=236.1648460182, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=) 2024-03-21T21:28:53.994 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:53 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:53.995 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:53 smithi066 podman[50813]: 2024-03-21 21:28:53.536383032 +0000 UTC m=+0.608930853 container remove a2c368760f2573062c039efb245cccfa316fa87b82712d6acf1fbbf85214a2fe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, vcs-type=git, description=Ceph Grafana Container, io.openshift.tags=base rhel8, version=8.5, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-21T21:28:53.995 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:53 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T21:28:53.995 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:28:53 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T21:28:54.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:53 smithi066 ceph-mon[25664]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:53 smithi028 ceph-mon[29568]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:54.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:53 smithi028 ceph-mon[24806]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:55.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:55 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:55.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:55 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:55.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:55 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:28:56.408 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:56 smithi066 ceph-mon[25664]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:56.409 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:56 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:56.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:56 smithi028 ceph-mon[24806]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:56.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:56 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:56.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:56 smithi028 ceph-mon[29568]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:56.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:56 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:28:58.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:58 smithi028 ceph-mon[29568]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:58.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:58 smithi028 ceph-mon[24806]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:58.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:58 smithi066 ceph-mon[25664]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:28:58.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:28:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:28:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:28:59.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:28:59 smithi028 ceph-mon[29568]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:59.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:28:59 smithi028 ceph-mon[24806]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:28:59.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:28:59 smithi066 ceph-mon[25664]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:02.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:01 smithi028 ceph-mon[29568]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:02.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:01 smithi028 ceph-mon[24806]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:02.247 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:01 smithi066 ceph-mon[25664]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:02.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:29:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:29:02] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-21T21:29:03.759 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:03 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-03-21T21:29:03.759 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:03 smithi066 systemd[1]: Stopped Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:29:03.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:04.026 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:03 smithi066 ceph-mon[25664]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:04.027 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:03 smithi066 systemd[1]: Starting Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:29:04.027 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:03 smithi066 podman[50893]: 2024-03-21 21:29:03.891503442 +0000 UTC m=+0.018862137 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T21:29:04.279 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 podman[50893]: 2024-03-21 21:29:04.028777212 +0000 UTC m=+0.156135899 container create 9d1b61ee42c449a1ea708bef0bb1632c34f783efbc87e86704dab2014bb5e8f2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, vendor=Red Hat, Inc., release=236.1648460182, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, architecture=x86_64, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-21T21:29:04.279 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 podman[50893]: 2024-03-21 21:29:04.250167501 +0000 UTC m=+0.377526201 container init 9d1b61ee42c449a1ea708bef0bb1632c34f783efbc87e86704dab2014bb5e8f2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, io.buildah.version=1.24.2, vcs-type=git, com.redhat.component=ubi8-container, distribution-scope=public, name=ubi8, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, maintainer=Paul Cuzner , description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-21T21:29:04.279 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 podman[50893]: 2024-03-21 21:29:04.253208279 +0000 UTC m=+0.380566968 container start 9d1b61ee42c449a1ea708bef0bb1632c34f783efbc87e86704dab2014bb5e8f2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, io.buildah.version=1.24.2, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, description=Ceph Grafana Container, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, vcs-type=git, build-date=2022-03-28T10:36:18.413762, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8) 2024-03-21T21:29:04.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:03 smithi028 ceph-mon[29568]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:04.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:03 smithi028 ceph-mon[24806]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:04.610 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[50907]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:29:04.610 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[50907]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T21:29:04.610 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 bash[50893]: 9d1b61ee42c449a1ea708bef0bb1632c34f783efbc87e86704dab2014bb5e8f2 2024-03-21T21:29:04.610 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 systemd[1]: Started Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:29:04.610 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 podman[50923]: 2024-03-21 21:29:04.341292164 +0000 UTC m=+0.053112179 container died 9d1b61ee42c449a1ea708bef0bb1632c34f783efbc87e86704dab2014bb5e8f2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., version=8.5, architecture=x86_64, description=Ceph Grafana Container, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, vcs-type=git, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-21T21:29:05.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:05 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:05.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 podman[50923]: 2024-03-21 21:29:04.936917285 +0000 UTC m=+0.648737301 container remove 9d1b61ee42c449a1ea708bef0bb1632c34f783efbc87e86704dab2014bb5e8f2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, version=8.5, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.buildah.version=1.24.2, release=236.1648460182, io.openshift.expose-services=, distribution-scope=public, maintainer=Paul Cuzner , name=ubi8, vcs-type=git, vendor=Red Hat, Inc.) 2024-03-21T21:29:05.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T21:29:05.248 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:04 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T21:29:05.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:05 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:05.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:05 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:06.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:06 smithi028 ceph-mon[24806]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:06.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:06 smithi028 ceph-mon[29568]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:06.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:06 smithi066 ceph-mon[25664]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:08.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:08 smithi028 ceph-mon[24806]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:08.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:08 smithi028 ceph-mon[29568]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:08.497 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:08 smithi066 ceph-mon[25664]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:08.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:08.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:09.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:09 smithi028 ceph-mon[24806]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:09.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:09 smithi028 ceph-mon[29568]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:09.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:09 smithi066 ceph-mon[25664]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:10.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:10 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:10.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:10 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:10.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:10 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:11.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:11 smithi028 ceph-mon[29568]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:11.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:11 smithi028 ceph-mon[24806]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:11.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:11 smithi066 ceph-mon[25664]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:12.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:29:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:29:12] "GET /metrics HTTP/1.1" 200 37956 "" "Prometheus/2.33.4" 2024-03-21T21:29:13.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:14.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:14 smithi028 ceph-mon[29568]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:14.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:14 smithi028 ceph-mon[24806]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:14.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:14 smithi066 ceph-mon[25664]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:15.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:15 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:15.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:15 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:15.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:15 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:15.498 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:15 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-03-21T21:29:15.498 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:15 smithi066 systemd[1]: Stopped Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:29:15.498 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:15 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-03-21T21:29:15.498 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:15 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T21:29:15.498 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:29:15 smithi066 systemd[1]: Failed to start Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:29:16.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:16 smithi028 ceph-mon[24806]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:29:16.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:16 smithi028 ceph-mon[29568]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:29:16.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:16 smithi066 ceph-mon[25664]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:29:18.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:18 smithi028 ceph-mon[24806]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:18.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:18 smithi028 ceph-mon[29568]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:18.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:18 smithi066 ceph-mon[25664]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:18.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:19.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:19 smithi028 ceph-mon[24806]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:29:19.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:19 smithi028 ceph-mon[29568]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:29:19.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:19 smithi066 ceph-mon[25664]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:29:22.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:22 smithi028 ceph-mon[24806]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:29:22.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:22 smithi028 ceph-mon[29568]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:29:22.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:22 smithi066 ceph-mon[25664]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:29:22.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:29:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:29:22] "GET /metrics HTTP/1.1" 200 37956 "" "Prometheus/2.33.4" 2024-03-21T21:29:23.873 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:24.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:24 smithi028 ceph-mon[24806]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:24.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:24 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:29:24.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:24 smithi028 ceph-mon[29568]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:24.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:24 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:29:24.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:24 smithi066 ceph-mon[25664]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:24.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:24 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:29:25.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:25 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:25.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:25 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:25.358 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:25 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:26.296 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:26 smithi066 ceph-mon[25664]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:26.296 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:26.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:26 smithi028 ceph-mon[29568]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:26.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:26.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:26 smithi028 ceph-mon[24806]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:26.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:28.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[24806]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:28.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:29:28.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:29:28.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[29568]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:28.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:29:28.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:29:28.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:27 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:27 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:27 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:27 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:27 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:27 smithi066 ceph-mon[25664]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:28.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:27 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:29:28.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:27 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:29:28.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:27 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:29:28.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:28 smithi066 ceph-mon[25664]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-21T21:29:28.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:28 smithi066 ceph-mon[25664]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:28.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:29.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:28 smithi028 ceph-mon[29568]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-21T21:29:29.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:28 smithi028 ceph-mon[29568]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:29.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:28 smithi028 ceph-mon[24806]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-21T21:29:29.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:28 smithi028 ceph-mon[24806]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:31.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:30 smithi028 ceph-mon[29568]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:31.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:30 smithi028 ceph-mon[24806]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:31.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:30 smithi066 ceph-mon[25664]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:32.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:32 smithi066 ceph-mon[25664]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-03-21T21:29:32.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:29:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:29:32] "GET /metrics HTTP/1.1" 200 37955 "" "Prometheus/2.33.4" 2024-03-21T21:29:33.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:32 smithi028 ceph-mon[29568]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-03-21T21:29:33.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:32 smithi028 ceph-mon[24806]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-03-21T21:29:33.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:34.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:34 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:34.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:34 smithi028 ceph-mon[29568]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:34.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:34 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:34.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:34 smithi028 ceph-mon[24806]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:34.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:34 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:34.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:34 smithi066 ceph-mon[25664]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:36 smithi066 ceph-mon[25664]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-03-21T21:29:36.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:36 smithi028 ceph-mon[24806]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-03-21T21:29:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:36 smithi028 ceph-mon[29568]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-03-21T21:29:38.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:38 smithi066 ceph-mon[25664]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:29:38.748 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:38.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:38 smithi028 ceph-mon[29568]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:29:38.864 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:38 smithi028 ceph-mon[24806]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:29:40.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:40 smithi066 ceph-mon[25664]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:40.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:40 smithi028 ceph-mon[29568]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:40.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:40 smithi028 ceph-mon[24806]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:41.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:41 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:41.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:41 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:41.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:41 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:42.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:42 smithi028 ceph-mon[24806]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:42.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:42 smithi028 ceph-mon[29568]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:42.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:42 smithi066 ceph-mon[25664]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:42.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:29:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:29:42] "GET /metrics HTTP/1.1" 200 37967 "" "Prometheus/2.33.4" 2024-03-21T21:29:43.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:44.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:44 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:44.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:44 smithi028 ceph-mon[24806]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:44 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:44 smithi028 ceph-mon[29568]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:44.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:44 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:44.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:44 smithi066 ceph-mon[25664]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:46.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:46 smithi066 ceph-mon[25664]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:46.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:46 smithi028 ceph-mon[24806]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:46.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:46 smithi028 ceph-mon[29568]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:48.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:48 smithi066 ceph-mon[25664]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:48.748 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:48 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:48.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:48 smithi028 ceph-mon[24806]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:48.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:48 smithi028 ceph-mon[29568]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:50.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:50 smithi066 ceph-mon[25664]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:50.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:50 smithi028 ceph-mon[24806]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:50.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:50 smithi028 ceph-mon[29568]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:52.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:52 smithi028 ceph-mon[24806]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:52.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:52 smithi028 ceph-mon[29568]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:52.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:52 smithi066 ceph-mon[25664]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:52.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:29:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:29:52] "GET /metrics HTTP/1.1" 200 37967 "" "Prometheus/2.33.4" 2024-03-21T21:29:53.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:53 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:29:54.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:54 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:54.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:54 smithi028 ceph-mon[24806]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:54.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:54 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:54.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:54 smithi028 ceph-mon[29568]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:54.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:54 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:29:54.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:54 smithi066 ceph-mon[25664]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:56.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:56 smithi066 ceph-mon[25664]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:56.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:56 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:56.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:56 smithi028 ceph-mon[24806]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:56.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:56 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:56.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:56 smithi028 ceph-mon[29568]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:29:56.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:56 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:29:58.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:29:58 smithi028 ceph-mon[29568]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:58.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:29:58 smithi028 ceph-mon[24806]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:58.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:29:58 smithi066 ceph-mon[25664]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:29:58.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:29:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:29:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:00.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:00 smithi028 ceph-mon[24806]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:00.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:00 smithi028 ceph-mon[24806]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T21:30:00.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:00 smithi028 ceph-mon[24806]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T21:30:00.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:00 smithi028 ceph-mon[24806]: daemon grafana.a on smithi066 is in error state 2024-03-21T21:30:00.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:00 smithi028 ceph-mon[29568]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:00.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:00 smithi028 ceph-mon[29568]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T21:30:00.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:00 smithi028 ceph-mon[29568]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T21:30:00.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:00 smithi028 ceph-mon[29568]: daemon grafana.a on smithi066 is in error state 2024-03-21T21:30:00.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:00 smithi066 ceph-mon[25664]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:00.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:00 smithi066 ceph-mon[25664]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T21:30:00.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:00 smithi066 ceph-mon[25664]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T21:30:00.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:00 smithi066 ceph-mon[25664]: daemon grafana.a on smithi066 is in error state 2024-03-21T21:30:02.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:02 smithi028 ceph-mon[24806]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:02.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:02 smithi028 ceph-mon[29568]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:02.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:02 smithi066 ceph-mon[25664]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:02.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:30:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:30:02] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-21T21:30:03.748 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:04.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:04 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:04.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:04 smithi028 ceph-mon[29568]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:04.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:04 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:04.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:04 smithi028 ceph-mon[24806]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:04.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:04 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:04.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:04 smithi066 ceph-mon[25664]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:06.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:06 smithi028 ceph-mon[29568]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:06.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:06 smithi028 ceph-mon[24806]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:06.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:06 smithi066 ceph-mon[25664]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:08.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:08 smithi028 ceph-mon[29568]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:08.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:08 smithi028 ceph-mon[24806]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:08.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:08 smithi066 ceph-mon[25664]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:08.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:10.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:10 smithi028 ceph-mon[29568]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:10.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:10 smithi028 ceph-mon[24806]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:10.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:10 smithi066 ceph-mon[25664]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:11.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:11 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:11.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:11 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:11.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:11 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:12.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:12 smithi028 ceph-mon[29568]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:12.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:12 smithi028 ceph-mon[24806]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:12.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:12 smithi066 ceph-mon[25664]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:12.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:30:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:30:12] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-03-21T21:30:13.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:14.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:14 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:14.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:14 smithi028 ceph-mon[29568]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:14.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:14 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:14.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:14 smithi028 ceph-mon[24806]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:14.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:14 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:14.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:14 smithi066 ceph-mon[25664]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:16.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:16 smithi028 ceph-mon[24806]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:16.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:16 smithi028 ceph-mon[29568]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:16.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:16 smithi066 ceph-mon[25664]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:18.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:18 smithi028 ceph-mon[24806]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:18.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:18 smithi028 ceph-mon[29568]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:18.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:18 smithi066 ceph-mon[25664]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:18.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:20.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:20 smithi028 ceph-mon[24806]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:20.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:20 smithi028 ceph-mon[29568]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:20.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:20 smithi066 ceph-mon[25664]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:22.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:22 smithi028 ceph-mon[24806]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:22.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:22 smithi028 ceph-mon[29568]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:22.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:22 smithi066 ceph-mon[25664]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:22.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:30:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:30:22] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-03-21T21:30:23.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:24.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:24 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:24.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:24 smithi028 ceph-mon[24806]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:24.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:24 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:24.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:24 smithi028 ceph-mon[29568]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:24.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:24 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:24.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:24 smithi066 ceph-mon[25664]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:26.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:26 smithi028 ceph-mon[24806]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:26.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:26.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:26 smithi028 ceph-mon[29568]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:26.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:26.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:26 smithi066 ceph-mon[25664]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:26.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:28.658 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:28 smithi066 ceph-mon[25664]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:28.658 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:28 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:30:28.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:28 smithi028 ceph-mon[24806]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:28.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:28 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:30:28.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:28 smithi028 ceph-mon[29568]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:28.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:28 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:30:28.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-21T21:30:28.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:30.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:30 smithi028 ceph-mon[24806]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:30.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:30 smithi028 ceph-mon[29568]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:30.838 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:30 smithi066 ceph-mon[25664]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:32.020 INFO:teuthology.orchestra.run.smithi028.stdout:Scheduled to redeploy mgr.y on host 'smithi028' 2024-03-21T21:30:32.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[24806]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3233885753' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[24806]: from='client.25075 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[24806]: Schedule redeploy daemon mgr.y 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[29568]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:32.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3233885753' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T21:30:32.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[29568]: from='client.25075 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:30:32.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[29568]: Schedule redeploy daemon mgr.y 2024-03-21T21:30:32.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:32 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-mon[25664]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:30:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3233885753' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T21:30:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-mon[25664]: from='client.25075 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:30:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-mon[25664]: Schedule redeploy daemon mgr.y 2024-03-21T21:30:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:32.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:30:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:30:32] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-21T21:30:33.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps --refresh' 2024-03-21T21:30:33.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:34.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:34 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:34.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:34 smithi028 ceph-mon[24806]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:34.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:34 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:34.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:34 smithi028 ceph-mon[29568]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:34.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:34 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:34.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:34 smithi066 ceph-mon[25664]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:30:35.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:35 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:35.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:35 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:35.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:35 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:35.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:35 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:30:35.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:35 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:30:35.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:35 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:35.766 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:30:35.766 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (2m) 1s ago 10m 25.7M - 0.23.0 ba2b418f427c cd065b97fad1 2024-03-21T21:30:35.766 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 error 4s ago 10m - - 2024-03-21T21:30:35.766 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (2m) 1s ago 9m 50.8M - 3.5 e1d6a67b021e 6f4a5e74673a 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283 running (5m) 4s ago 15m 513M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 7fe31a1b09b6 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:9283 running (17m) 1s ago 17m 411M - 17.2.0 e1d6a67b021e 602bce1ea97f 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (17m) 1s ago 17m 60.6M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (15m) 4s ago 15m 49.3M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (15m) 1s ago 15m 46.8M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (2m) 1s ago 11m 18.2M - 1.3.1 1dbe0e931976 a8a424c65ebe 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (2m) 4s ago 11m 17.7M - 1.3.1 1dbe0e931976 f6cdb9cb9923 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (15m) 1s ago 15m 51.5M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (14m) 1s ago 14m 52.4M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (14m) 1s ago 14m 50.7M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (13m) 1s ago 13m 51.1M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (13m) 4s ago 13m 51.2M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (13m) 4s ago 13m 50.2M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:30:35.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (12m) 4s ago 12m 49.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:30:35.768 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (12m) 4s ago 12m 50.5M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:30:35.768 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (2m) 4s ago 11m 71.5M - 2.33.4 514e6a882f6e cc8b8c475056 2024-03-21T21:30:35.768 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (10m) 1s ago 10m 73.3M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:30:35.768 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (9m) 4s ago 9m 71.8M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:30:36.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'sleep 180' 2024-03-21T21:30:36.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[29568]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[29568]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[29568]: Deploying daemon mgr.y on smithi028 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[29568]: from='client.25081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[24806]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[24806]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:30:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:30:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:30:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:30:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:30:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[24806]: Deploying daemon mgr.y on smithi028 2024-03-21T21:30:36.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:36 smithi028 ceph-mon[24806]: from='client.25081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:30:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:36 smithi066 ceph-mon[25664]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-21T21:30:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:36 smithi066 ceph-mon[25664]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:30:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:36 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:30:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:36 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:30:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:36 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:30:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:36 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:30:36.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:36 smithi066 ceph-mon[25664]: Deploying daemon mgr.y on smithi028 2024-03-21T21:30:36.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:36 smithi066 ceph-mon[25664]: from='client.25081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:30:38.462 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:38 smithi028 ceph-mon[29568]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-21T21:30:38.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:38 smithi028 ceph-mon[24806]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-21T21:30:38.658 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:38 smithi066 ceph-mon[25664]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-21T21:30:38.774 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:38 smithi028 systemd[1]: Stopping Ceph mgr.y for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:30:38.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:39.580 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:39 smithi028 podman[62314]: 2024-03-21 21:30:39.29587622 +0000 UTC m=+0.128062443 container died 602bce1ea97f017f8c18b99843a57d9c364f5a86b84a9bc195236bb02ff3f824 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, RELEASE=HEAD, distribution-scope=public, maintainer=Guillaume Abrioux , architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_CLEAN=True, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-21T21:30:39.580 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:39 smithi028 podman[62314]: 2024-03-21 21:30:39.581453431 +0000 UTC m=+0.413639650 container cleanup 602bce1ea97f017f8c18b99843a57d9c364f5a86b84a9bc195236bb02ff3f824 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.expose-services=, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-21T21:30:39.899 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:39 smithi028 bash[62314]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y 2024-03-21T21:30:39.899 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:39 smithi028 podman[62328]: 2024-03-21 21:30:39.799392214 +0000 UTC m=+0.505014336 container remove 602bce1ea97f017f8c18b99843a57d9c364f5a86b84a9bc195236bb02ff3f824 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, release=754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream) 2024-03-21T21:30:39.900 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:39 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:30:40.194 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:39 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-21T21:30:40.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:39 smithi028 systemd[1]: Stopped Ceph mgr.y for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:30:40.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:39 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.y.service: Consumed 1min 815ms CPU time. 2024-03-21T21:30:40.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:40 smithi028 systemd[1]: Starting Ceph mgr.y for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:30:40.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:40 smithi028 ceph-mon[29568]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-21T21:30:40.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:40 smithi028 ceph-mon[24806]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-21T21:30:40.602 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:40 smithi028 podman[62459]: 2024-03-21 21:30:40.317442176 +0000 UTC m=+0.107678489 container create 675d4e55bc325e278e88ae24f9e45c32a098bd9829dda310f092518434231616 (image=quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-21T21:30:40.602 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:40 smithi028 podman[62459]: 2024-03-21 21:30:40.228248961 +0000 UTC m=+0.018485241 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:30:40.602 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:40 smithi028 podman[62459]: 2024-03-21 21:30:40.536910848 +0000 UTC m=+0.327147122 container init 675d4e55bc325e278e88ae24f9e45c32a098bd9829dda310f092518434231616 (image=quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-21T21:30:40.602 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:40 smithi028 podman[62459]: 2024-03-21 21:30:40.5395545 +0000 UTC m=+0.329790780 container start 675d4e55bc325e278e88ae24f9e45c32a098bd9829dda310f092518434231616 (image=quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:30:40.602 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:40 smithi028 ceph-mgr[62478]: -- 172.21.15.28:0/2870501085 <== mon.1 v2:172.21.15.28:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5618785ec5a0 con 0x5618785c1000 2024-03-21T21:30:40.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:40 smithi066 ceph-mon[25664]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-21T21:30:40.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:40 smithi028 bash[62459]: 675d4e55bc325e278e88ae24f9e45c32a098bd9829dda310f092518434231616 2024-03-21T21:30:40.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:40 smithi028 systemd[1]: Started Ceph mgr.y for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:30:41.444 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:41.126+0000 7fbffb334200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:30:41.445 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:41.236+0000 7fbffb334200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:30:41.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:41.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:30:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:30:41.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:41 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:41.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:41 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:41 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:41 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:41 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:41.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:41 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:30:41.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:41.925+0000 7fbffb334200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:30:42.294 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:42.028+0000 7fbffb334200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:30:42.294 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:42.181+0000 7fbffb334200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:30:42.294 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:42.294+0000 7fbffb334200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:30:42.677 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:42.394+0000 7fbffb334200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:30:42.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-mon[24806]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s 2024-03-21T21:30:42.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-mon[29568]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s 2024-03-21T21:30:42.945 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:42.678+0000 7fbffb334200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T21:30:42.945 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:42.799+0000 7fbffb334200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:30:42.945 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:42.892+0000 7fbffb334200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:30:42.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:42 smithi066 ceph-mon[25664]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s 2024-03-21T21:30:42.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:30:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:30:42] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-03-21T21:30:43.440 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:42 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:42.994+0000 7fbffb334200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:30:43.440 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:43.177+0000 7fbffb334200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:30:43.794 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:44.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:43 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:44.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:43 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:44.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:43 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:44.544 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:44.231+0000 7fbffb334200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:30:44.544 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:44.369+0000 7fbffb334200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:30:44.795 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:44.718+0000 7fbffb334200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:30:45.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:44 smithi028 ceph-mon[24806]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-21T21:30:45.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:44 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:45.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:44 smithi028 ceph-mon[29568]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-21T21:30:45.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:44 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:45.047 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:44 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:44.823+0000 7fbffb334200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:30:45.047 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:45.046+0000 7fbffb334200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:30:45.109 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:44 smithi066 ceph-mon[25664]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-03-21T21:30:45.109 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:44 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:45.404 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:45.241+0000 7fbffb334200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:30:45.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:45.546+0000 7fbffb334200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:30:46.071 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:45.820+0000 7fbffb334200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:30:46.444 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:46.072+0000 7fbffb334200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:30:46.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:46 smithi028 ceph-mon[29568]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:46.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:46 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:46.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:46 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:46.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:46 smithi028 ceph-mon[24806]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:46.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:46 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:46.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:46 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:47.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:46 smithi066 ceph-mon[25664]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:47.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:46 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:47.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:46 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:47.444 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:46 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:46.946+0000 7fbffb334200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:30:47.879 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:47.545+0000 7fbffb334200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:30:47.879 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:47.648+0000 7fbffb334200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:30:47.879 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:47.879+0000 7fbffb334200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:30:48.173 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:30:47] ENGINE Bus STARTING 2024-03-21T21:30:48.173 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CherryPy Checker: 2024-03-21T21:30:48.173 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: The Application mounted at '' has an empty config. 2024-03-21T21:30:48.173 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:30:48.173 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:30:47] ENGINE Serving on http://:::9283 2024-03-21T21:30:48.173 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:30:47] ENGINE Bus STARTED 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: Standby manager daemon y restarted 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: Standby manager daemon y started 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T21:30:48.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: Standby manager daemon y restarted 2024-03-21T21:30:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: Standby manager daemon y started 2024-03-21T21:30:48.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T21:30:48.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:30:48.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T21:30:48.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:48 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:30:48.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: Standby manager daemon y restarted 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: Standby manager daemon y started 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T21:30:48.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:30:48.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:48 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:49.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:49 smithi028 ceph-mon[24806]: mgrmap e26: x(active, since 3m), standbys: y 2024-03-21T21:30:49.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:49 smithi028 ceph-mon[29568]: mgrmap e26: x(active, since 3m), standbys: y 2024-03-21T21:30:49.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:49 smithi066 ceph-mon[25664]: mgrmap e26: x(active, since 3m), standbys: y 2024-03-21T21:30:50.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:50 smithi028 ceph-mon[24806]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:50.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:50 smithi028 ceph-mon[29568]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:50.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:50 smithi066 ceph-mon[25664]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:52.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:52 smithi028 ceph-mon[29568]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:52.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:52 smithi028 ceph-mon[24806]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:52.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:52 smithi066 ceph-mon[25664]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:52.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:30:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:30:52] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-03-21T21:30:53.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:53 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:30:54.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:54 smithi028 ceph-mon[29568]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:54.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:54 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:54.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:54 smithi028 ceph-mon[24806]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:54.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:54 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:54.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:54 smithi066 ceph-mon[25664]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:54.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:54 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:30:56.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:56 smithi028 ceph-mon[29568]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:56.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:56 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:56.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:56 smithi028 ceph-mon[24806]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:56.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:56 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:56.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:56 smithi066 ceph-mon[25664]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-21T21:30:56.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:56 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:30:58.658 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:30:58 smithi066 ceph-mon[25664]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:30:58.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:30:58 smithi028 ceph-mon[29568]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:30:58.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:30:58 smithi028 ceph-mon[24806]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:30:58.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:30:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:30:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:00.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:00 smithi028 ceph-mon[24806]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:00.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:00 smithi028 ceph-mon[29568]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:00.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:00 smithi066 ceph-mon[25664]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:02.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:02 smithi028 ceph-mon[24806]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:02.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:02 smithi028 ceph-mon[29568]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:02.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:02 smithi066 ceph-mon[25664]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:02.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:31:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:31:02] "GET /metrics HTTP/1.1" 200 37960 "" "Prometheus/2.33.4" 2024-03-21T21:31:03.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:04.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:04 smithi028 ceph-mon[24806]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:04.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:04 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:04.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:04 smithi028 ceph-mon[29568]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:04.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:04 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:04.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:04 smithi066 ceph-mon[25664]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:04.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:04 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:06.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:06 smithi028 ceph-mon[24806]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:06.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:06 smithi028 ceph-mon[29568]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:06.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:06 smithi066 ceph-mon[25664]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:08.498 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:08 smithi066 ceph-mon[25664]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:08.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:08 smithi028 ceph-mon[24806]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:08.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:08 smithi028 ceph-mon[29568]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:08.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:10.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:10 smithi028 ceph-mon[24806]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:10.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:10 smithi028 ceph-mon[29568]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:10.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:10 smithi066 ceph-mon[25664]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:11.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:11 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:11.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:11 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:11.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:11 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:12 smithi028 ceph-mon[24806]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:12.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:12 smithi028 ceph-mon[29568]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:12.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:12 smithi066 ceph-mon[25664]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:12.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:31:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:31:12] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-21T21:31:13.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:14.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:14 smithi028 ceph-mon[29568]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:14.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:14 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:14.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:14 smithi028 ceph-mon[24806]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:14.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:14 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:14.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:14 smithi066 ceph-mon[25664]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:14.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:14 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:16.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:16 smithi028 ceph-mon[24806]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:16.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:16 smithi028 ceph-mon[29568]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:16.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:16 smithi066 ceph-mon[25664]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:18.658 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:18 smithi066 ceph-mon[25664]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:18.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:18 smithi028 ceph-mon[24806]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:18.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:18 smithi028 ceph-mon[29568]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:18.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:20.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:20 smithi028 ceph-mon[29568]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:20.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:20 smithi028 ceph-mon[24806]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:20.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:20 smithi066 ceph-mon[25664]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:22.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:22 smithi028 ceph-mon[29568]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:22.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:22 smithi028 ceph-mon[24806]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:22.747 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:22 smithi066 ceph-mon[25664]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:22.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:31:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:31:22] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-21T21:31:23.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:24.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:24 smithi028 ceph-mon[29568]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:24.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:24 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:24.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:24 smithi028 ceph-mon[24806]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:24.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:24 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:24.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:24 smithi066 ceph-mon[25664]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:24.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:24 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:26.552 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:26 smithi066 ceph-mon[25664]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:26.553 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:26 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:26.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:26 smithi028 ceph-mon[29568]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:26.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:26 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:26.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:26 smithi028 ceph-mon[24806]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:26.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:26 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:28.658 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:28 smithi066 ceph-mon[25664]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:28.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:28 smithi028 ceph-mon[29568]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:28.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:28 smithi028 ceph-mon[24806]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:28.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:30.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:30 smithi028 ceph-mon[29568]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:30.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:30 smithi028 ceph-mon[24806]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:30.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:30 smithi066 ceph-mon[25664]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:32.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:32 smithi028 ceph-mon[29568]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:32.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:32 smithi028 ceph-mon[24806]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:32.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:32 smithi066 ceph-mon[25664]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:32.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:31:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:31:32] "GET /metrics HTTP/1.1" 200 37959 "" "Prometheus/2.33.4" 2024-03-21T21:31:33.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:34.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:34 smithi028 ceph-mon[29568]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:34.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:34 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:34.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:34 smithi028 ceph-mon[24806]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:34.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:34 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:34.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:34 smithi066 ceph-mon[25664]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:34.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:34 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:36.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:36 smithi028 ceph-mon[24806]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:36.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:36 smithi028 ceph-mon[29568]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:36.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:36 smithi066 ceph-mon[25664]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:38.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:38 smithi028 ceph-mon[24806]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:38 smithi028 ceph-mon[29568]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:38.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:38 smithi066 ceph-mon[25664]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:38.749 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:40.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:40 smithi028 ceph-mon[24806]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:40.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:40 smithi028 ceph-mon[29568]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:40.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:40 smithi066 ceph-mon[25664]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:41.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:41 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:41.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:41 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:41.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:41 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:42.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:42 smithi028 ceph-mon[24806]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:42.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:42 smithi028 ceph-mon[29568]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:42.746 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:42 smithi066 ceph-mon[25664]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:42.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:31:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:31:42] "GET /metrics HTTP/1.1" 200 37961 "" "Prometheus/2.33.4" 2024-03-21T21:31:43.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:44 smithi028 ceph-mon[29568]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:44 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:44.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:44 smithi028 ceph-mon[24806]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:44.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:44 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:44.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:44 smithi066 ceph-mon[25664]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:44.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:44 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:46.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:46 smithi028 ceph-mon[29568]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:46.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:46 smithi028 ceph-mon[24806]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:46.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:46 smithi066 ceph-mon[25664]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:47.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:47 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:31:47.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:47 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:31:47.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:47 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:31:48.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:48 smithi028 ceph-mon[29568]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:48.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:48 smithi028 ceph-mon[24806]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:48.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:48 smithi066 ceph-mon[25664]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:31:48.749 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:48 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:50.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:50 smithi028 ceph-mon[24806]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:50.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:50 smithi028 ceph-mon[29568]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:50.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:50 smithi066 ceph-mon[25664]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:52.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[29568]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:52.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:31:52.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:31:52.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:31:52.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:31:52.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:31:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[24806]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:31:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:31:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:31:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:31:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:52 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:31:52.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:31:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:31:52] "GET /metrics HTTP/1.1" 200 37961 "" "Prometheus/2.33.4" 2024-03-21T21:31:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:52 smithi066 ceph-mon[25664]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:31:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:52 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:31:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:52 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:31:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:52 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:31:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:52 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:31:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:52 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:31:53.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:53 smithi028 ceph-mon[29568]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T21:31:53.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:53 smithi028 ceph-mon[29568]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:31:53.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:53 smithi028 ceph-mon[24806]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T21:31:53.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:53 smithi028 ceph-mon[24806]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:31:53.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:53 smithi066 ceph-mon[25664]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T21:31:53.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:53 smithi066 ceph-mon[25664]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:31:53.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:53 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:54.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:54 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:54.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:54 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:54.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:54 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:31:55.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:55 smithi028 ceph-mon[24806]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:31:55.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:55 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:55.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:55 smithi028 ceph-mon[29568]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:31:55.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:55 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:55.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:55 smithi066 ceph-mon[25664]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:31:55.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:55 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:31:57.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:57 smithi028 ceph-mon[24806]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-03-21T21:31:57.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:57 smithi028 ceph-mon[29568]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-03-21T21:31:57.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:57 smithi066 ceph-mon[25664]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-03-21T21:31:58.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:31:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:31:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:31:59.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:31:59 smithi066 ceph-mon[25664]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:00.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:31:59 smithi028 ceph-mon[24806]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:00.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:31:59 smithi028 ceph-mon[29568]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:01.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:01 smithi066 ceph-mon[25664]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:02.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:01 smithi028 ceph-mon[24806]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:02.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:01 smithi028 ceph-mon[29568]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:02.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:32:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:32:02] "GET /metrics HTTP/1.1" 200 37957 "" "Prometheus/2.33.4" 2024-03-21T21:32:03.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:03 smithi028 ceph-mon[24806]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:32:03.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:03 smithi028 ceph-mon[29568]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:32:03.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:03 smithi066 ceph-mon[25664]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:32:03.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:04.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:04 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:04 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:05.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:04 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:05.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:05 smithi066 ceph-mon[25664]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:06.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:05 smithi028 ceph-mon[24806]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:06.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:05 smithi028 ceph-mon[29568]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:07.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:07 smithi066 ceph-mon[25664]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:08.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:07 smithi028 ceph-mon[24806]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:08.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:07 smithi028 ceph-mon[29568]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:08.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:09.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:09 smithi066 ceph-mon[25664]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:10.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:09 smithi028 ceph-mon[24806]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:10.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:09 smithi028 ceph-mon[29568]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:10.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:10 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:11.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:10 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:11.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:10 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:11.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:11 smithi066 ceph-mon[25664]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:12.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:11 smithi028 ceph-mon[24806]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:12.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:11 smithi028 ceph-mon[29568]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:12.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:32:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:32:12] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-21T21:32:13.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:13 smithi028 ceph-mon[24806]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:13.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:13 smithi028 ceph-mon[29568]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:13.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:13 smithi066 ceph-mon[25664]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:13.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:14.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:14 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:15.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:14 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:15.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:14 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:15.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:15 smithi066 ceph-mon[25664]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:16.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:15 smithi028 ceph-mon[24806]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:16.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:15 smithi028 ceph-mon[29568]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:17.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:17 smithi066 ceph-mon[25664]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:18.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:17 smithi028 ceph-mon[24806]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:18.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:17 smithi028 ceph-mon[29568]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:18.998 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:19.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:19 smithi066 ceph-mon[25664]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:20.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:19 smithi028 ceph-mon[24806]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:20.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:19 smithi028 ceph-mon[29568]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:21.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:21 smithi066 ceph-mon[25664]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:22.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:21 smithi028 ceph-mon[24806]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:22.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:21 smithi028 ceph-mon[29568]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:22.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:32:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:32:22] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-21T21:32:23.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:23 smithi028 ceph-mon[24806]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:23.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:23 smithi028 ceph-mon[29568]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:23.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:23 smithi066 ceph-mon[25664]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:23.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:24.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:24 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:25.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:24 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:25.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:24 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:25.998 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:25 smithi066 ceph-mon[25664]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:25.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:25 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:26.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:25 smithi028 ceph-mon[24806]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:26.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:25 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:25 smithi028 ceph-mon[29568]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:25 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:28.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:27 smithi028 ceph-mon[24806]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:28.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:27 smithi028 ceph-mon[29568]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:28.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:27 smithi066 ceph-mon[25664]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:28.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:30.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:29 smithi028 ceph-mon[29568]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:30.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:29 smithi028 ceph-mon[24806]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:30.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:29 smithi066 ceph-mon[25664]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:32.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:31 smithi028 ceph-mon[24806]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:32.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:31 smithi028 ceph-mon[29568]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:32.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:31 smithi066 ceph-mon[25664]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:32.998 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:32:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:32:32] "GET /metrics HTTP/1.1" 200 37959 "" "Prometheus/2.33.4" 2024-03-21T21:32:33.759 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:33.508Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:33.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:33 smithi028 ceph-mon[24806]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:33 smithi028 ceph-mon[29568]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:34.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:33 smithi066 ceph-mon[25664]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:35.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:34 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:34 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:35.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:34 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:36.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:35 smithi028 ceph-mon[24806]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:36.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:35 smithi028 ceph-mon[29568]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:36.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:35 smithi066 ceph-mon[25664]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:38.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:37 smithi028 ceph-mon[24806]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:38.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:37 smithi028 ceph-mon[29568]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:38.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:37 smithi066 ceph-mon[25664]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:38.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:40.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:39 smithi028 ceph-mon[24806]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:40.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:39 smithi028 ceph-mon[29568]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:40.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:39 smithi066 ceph-mon[25664]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:40.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:40 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:40.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:40 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:41.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:40 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:42.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:41 smithi028 ceph-mon[24806]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:42.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:41 smithi028 ceph-mon[29568]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:42.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:41 smithi066 ceph-mon[25664]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:42.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:32:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:32:42] "GET /metrics HTTP/1.1" 200 37959 "" "Prometheus/2.33.4" 2024-03-21T21:32:43.778 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:43 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:43.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:43 smithi028 ceph-mon[29568]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:43.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:43 smithi028 ceph-mon[24806]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:44.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:43 smithi066 ceph-mon[25664]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:45.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:44 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:45.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:44 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:45.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:44 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:46.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:45 smithi028 ceph-mon[29568]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:46.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:45 smithi028 ceph-mon[24806]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:46.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:45 smithi066 ceph-mon[25664]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:48.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:47 smithi028 ceph-mon[29568]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:48.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:47 smithi028 ceph-mon[24806]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:48.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:47 smithi066 ceph-mon[25664]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:48.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:48 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:50.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:49 smithi028 ceph-mon[29568]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:50.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:49 smithi028 ceph-mon[24806]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:50.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:49 smithi066 ceph-mon[25664]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:52.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:51 smithi028 ceph-mon[29568]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:52.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:51 smithi028 ceph-mon[24806]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:52.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:51 smithi066 ceph-mon[25664]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:32:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:52 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:32:52.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:32:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:32:52] "GET /metrics HTTP/1.1" 200 37959 "" "Prometheus/2.33.4" 2024-03-21T21:32:53.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:52 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:32:53.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:52 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:32:53.802 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:53 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:53.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:53 smithi028 ceph-mon[29568]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:53.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:53 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:32:53.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:53 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:32:53.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:53 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:32:53.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:53 smithi028 ceph-mon[24806]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:53.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:53 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:32:53.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:53 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:32:53.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:53 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:32:54.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:53 smithi066 ceph-mon[25664]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:32:54.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:53 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:32:54.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:53 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:32:54.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:53 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:32:55.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:54 smithi028 ceph-mon[24806]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-21T21:32:55.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:54 smithi028 ceph-mon[24806]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:55.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:54 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:55.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:54 smithi028 ceph-mon[29568]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-21T21:32:55.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:54 smithi028 ceph-mon[29568]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:55.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:54 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:55.248 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:54 smithi066 ceph-mon[25664]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-21T21:32:55.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:54 smithi066 ceph-mon[25664]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:55.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:54 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:32:56.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:55 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:56.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:55 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:56.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:55 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:32:57.150 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:56 smithi066 ceph-mon[25664]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:57.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:56 smithi028 ceph-mon[24806]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:57.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:56 smithi028 ceph-mon[29568]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:58.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:32:58 smithi066 ceph-mon[25664]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:58.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:32:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:32:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"172.21.15.66:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.28\", device_class=\"ssd\", hostname=\"smithi028\", instance=\"smithi066:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.28\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:32:59.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:32:58 smithi028 ceph-mon[24806]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:32:59.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:32:58 smithi028 ceph-mon[29568]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:33:01.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:00 smithi028 ceph-mon[24806]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:33:01.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:00 smithi028 ceph-mon[29568]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:33:01.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:00 smithi066 ceph-mon[25664]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:33:02.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:02 smithi066 ceph-mon[25664]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-21T21:33:02.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:33:02] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-21T21:33:03.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:02 smithi028 ceph-mon[24806]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-21T21:33:03.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:02 smithi028 ceph-mon[29568]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-21T21:33:03.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:33:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:33:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi028\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"c10cc158-e7c7-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi028\", job=\"node\", machine=\"x86_64\", nodename=\"smithi028\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T21:33:05.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:04 smithi028 ceph-mon[29568]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:33:05.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:04 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:05.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:04 smithi028 ceph-mon[24806]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:33:05.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:04 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:05.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:04 smithi066 ceph-mon[25664]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:33:05.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:04 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:07.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:06 smithi028 ceph-mon[29568]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:07.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:06 smithi028 ceph-mon[24806]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:07.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:06 smithi066 ceph-mon[25664]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:09.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:08 smithi028 ceph-mon[24806]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:09.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:08 smithi028 ceph-mon[29568]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:09.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:08 smithi066 ceph-mon[25664]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:11.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:10 smithi028 ceph-mon[24806]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:11.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:10 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:11.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:10 smithi028 ceph-mon[29568]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:11.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:10 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:11.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:10 smithi066 ceph-mon[25664]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:11.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:10 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:12.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:12 smithi066 ceph-mon[25664]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:12.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:33:12] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-21T21:33:13.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:12 smithi028 ceph-mon[24806]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:13.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:12 smithi028 ceph-mon[29568]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:15.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:14 smithi028 ceph-mon[24806]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:15.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:14 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:15.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:14 smithi028 ceph-mon[29568]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:15.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:14 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:15.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:14 smithi066 ceph-mon[25664]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:15.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:14 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:16 smithi028 ceph-mon[24806]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:17.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:16 smithi028 ceph-mon[29568]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:17.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:16 smithi066 ceph-mon[25664]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:19.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:18 smithi028 ceph-mon[24806]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:19.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:18 smithi028 ceph-mon[29568]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:19.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:18 smithi066 ceph-mon[25664]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:20 smithi028 ceph-mon[24806]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:20 smithi028 ceph-mon[29568]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:21.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:20 smithi066 ceph-mon[25664]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:22.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:22 smithi066 ceph-mon[25664]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:22.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:33:22] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-21T21:33:23.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:22 smithi028 ceph-mon[24806]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:23.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:22 smithi028 ceph-mon[29568]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:25.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:24 smithi028 ceph-mon[29568]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:25.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:24 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:25.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:24 smithi028 ceph-mon[24806]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:25.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:24 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:25.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:24 smithi066 ceph-mon[25664]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:25.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:24 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:25 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:25 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:26.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:25 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:27.150 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:26 smithi066 ceph-mon[25664]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:27.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:26 smithi028 ceph-mon[29568]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:27.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:26 smithi028 ceph-mon[24806]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:29.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:28 smithi028 ceph-mon[29568]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:29.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:28 smithi028 ceph-mon[24806]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:29.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:28 smithi066 ceph-mon[25664]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:31.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:30 smithi028 ceph-mon[29568]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:31.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:30 smithi028 ceph-mon[24806]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:31.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:30 smithi066 ceph-mon[25664]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:32.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:32 smithi066 ceph-mon[25664]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:32.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:33:32] "GET /metrics HTTP/1.1" 200 37957 "" "Prometheus/2.33.4" 2024-03-21T21:33:33.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:32 smithi028 ceph-mon[24806]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:33.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:32 smithi028 ceph-mon[29568]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:34 smithi028 ceph-mon[29568]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:35.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:34 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:35.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:34 smithi028 ceph-mon[24806]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:35.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:34 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:35.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:34 smithi066 ceph-mon[25664]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:35.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:34 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:37.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:36 smithi028 ceph-mon[29568]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:37.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:36 smithi028 ceph-mon[24806]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:37.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:36 smithi066 ceph-mon[25664]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:39.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:38 smithi028 ceph-mon[24806]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:39.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:38 smithi028 ceph-mon[29568]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:39.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:38 smithi066 ceph-mon[25664]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:39.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:33:41.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:40 smithi028 ceph-mon[24806]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:41.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:40 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:41.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:40 smithi028 ceph-mon[29568]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:41.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:40 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:41.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:40 smithi066 ceph-mon[25664]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:41.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:40 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:42.018 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:33:42.018 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (5m) 110s ago 13m 23.8M - 0.23.0 ba2b418f427c cd065b97fad1 2024-03-21T21:33:42.018 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 error 2m ago 13m - - 2024-03-21T21:33:42.018 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (5m) 110s ago 12m 51.1M - 3.5 e1d6a67b021e 6f4a5e74673a 2024-03-21T21:33:42.018 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283 running (8m) 2m ago 18m 513M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 7fe31a1b09b6 2024-03-21T21:33:42.018 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (3m) 110s ago 20m 452M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:33:42.018 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (20m) 110s ago 20m 61.7M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (18m) 2m ago 18m 50.0M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (18m) 110s ago 18m 44.8M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (6m) 110s ago 14m 17.5M - 1.3.1 1dbe0e931976 a8a424c65ebe 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (5m) 2m ago 14m 17.2M - 1.3.1 1dbe0e931976 f6cdb9cb9923 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (18m) 110s ago 18m 52.4M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (17m) 110s ago 17m 53.4M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (17m) 110s ago 17m 52.0M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (16m) 110s ago 16m 51.3M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (16m) 2m ago 16m 51.7M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (16m) 2m ago 16m 50.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (15m) 2m ago 15m 50.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:33:42.019 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (15m) 2m ago 15m 51.4M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:33:42.020 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (5m) 2m ago 14m 71.5M - 2.33.4 514e6a882f6e cc8b8c475056 2024-03-21T21:33:42.020 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (13m) 110s ago 13m 74.0M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:33:42.020 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (13m) 2m ago 13m 71.9M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:33:42.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions' 2024-03-21T21:33:42.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:42 smithi066 ceph-mon[25664]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:42.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:42 smithi066 ceph-mon[25664]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:33:42.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:33:42] "GET /metrics HTTP/1.1" 200 37959 "" "Prometheus/2.33.4" 2024-03-21T21:33:43.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:42 smithi028 ceph-mon[24806]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:43.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:42 smithi028 ceph-mon[24806]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:33:43.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:42 smithi028 ceph-mon[29568]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:43.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:42 smithi028 ceph-mon[29568]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:33:44.877 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:33:44.877 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:33:44.877 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T21:33:44.878 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:33:44.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:33:44.879 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:33:45.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:44 smithi028 ceph-mon[29568]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:45.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:44 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:44 smithi028 ceph-mon[24806]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:44 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:45.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:44 smithi066 ceph-mon[25664]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:45.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:44 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:33:45.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:33:46.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:45 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2905537619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:33:46.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:45 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2905537619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:33:46.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:45 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2905537619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:33:47.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:46 smithi028 ceph-mon[29568]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:47.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:46 smithi028 ceph-mon[24806]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:47.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:46 smithi066 ceph-mon[25664]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:47.744 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T21:33:47.744 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T21:33:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: daemon grafana.a on smithi066 is in error state 2024-03-21T21:33:48.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:47 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3635890770' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:33:48.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:47 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3635890770' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:33:48.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:47 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3635890770' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:33:48.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph -s' 2024-03-21T21:33:48.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:48 smithi028 ceph-mon[24806]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:48.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:48 smithi028 ceph-mon[29568]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:49.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:48 smithi066 ceph-mon[25664]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:33:50.490 INFO:teuthology.orchestra.run.smithi028.stdout: cluster: 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: id: c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: health: HEALTH_WARN 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: 1 failed cephadm daemon(s) 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: services: 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: mgr: x(active, since 6m), standbys: y 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: data: 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: pools: 6 pools, 161 pgs 2024-03-21T21:33:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: objects: 209 objects, 585 KiB 2024-03-21T21:33:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: usage: 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: pgs: 161 active+clean 2024-03-21T21:33:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:33:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: io: 2024-03-21T21:33:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-21T21:33:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:33:51.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:50 smithi028 ceph-mon[29568]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:51.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:50 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3082011482' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:33:51.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:50 smithi028 ceph-mon[24806]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:51.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:50 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3082011482' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:33:51.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph mgr fail' 2024-03-21T21:33:51.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:50 smithi066 ceph-mon[25664]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:51.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:50 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3082011482' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:33:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:52 smithi066 ceph-mon[25664]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:52.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:33:52] "GET /metrics HTTP/1.1" 200 37959 "" "Prometheus/2.33.4" 2024-03-21T21:33:53.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:52 smithi028 ceph-mon[29568]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:53.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:52 smithi028 ceph-mon[24806]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:33:54.155 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:33:54.155 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/879164565' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:33:54.156 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-mon[25664]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:33:54.156 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-mon[25664]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T21:33:54.156 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:33:54.156 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-mon[25664]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:33:54.156 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-mon[25664]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:33:54.156 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:53.904+0000 7f8f57ae5700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-21T21:33:54.156 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ignoring --setuser ceph since I am not root 2024-03-21T21:33:54.156 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ignoring --setgroup ceph since I am not root 2024-03-21T21:33:54.156 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:53 smithi066 ceph-mgr[45923]: -- 172.21.15.66:0/620917982 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580213f25a0 con 0x5580213c7000 2024-03-21T21:33:54.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:33:54.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/879164565' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:33:54.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:33:54.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[29568]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T21:33:54.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:33:54.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[29568]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:33:54.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[29568]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:33:54.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:33:53] ENGINE Bus STOPPING 2024-03-21T21:33:54.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:33:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:33:54.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:33:53] ENGINE Bus STOPPED 2024-03-21T21:33:54.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:33:54] ENGINE Bus STARTING 2024-03-21T21:33:54.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:33:54] ENGINE Serving on http://:::9283 2024-03-21T21:33:54.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:33:54] ENGINE Bus STARTED 2024-03-21T21:33:54.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:33:54.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/879164565' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:33:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[24806]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:33:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[24806]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T21:33:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:33:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[24806]: from='mgr.34811 172.21.15.66:0/4128689984' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:33:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:53 smithi028 ceph-mon[24806]: from='mgr.34811 ' entity='mgr.x' 2024-03-21T21:33:54.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'sleep 180' 2024-03-21T21:33:54.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:54.498+0000 7fe22687c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:33:54.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:54.598+0000 7fe22687c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: mgrmap e27: y(active, starting, since 0.733964s) 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:33:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: Manager daemon y is now available 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[29568]: [21/Mar/2024:21:33:54] ENGINE Bus STARTING 2024-03-21T21:33:54.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: mgrmap e27: y(active, starting, since 0.733964s) 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:33:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:33:54.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: Manager daemon y is now available 2024-03-21T21:33:54.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:33:54.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:54.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:33:54.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:33:54.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:33:54.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:33:54.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:54 smithi028 ceph-mon[24806]: [21/Mar/2024:21:33:54] ENGINE Bus STARTING 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: mgrmap e27: y(active, starting, since 0.733964s) 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:33:55.163 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: Manager daemon y is now available 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:33:55.164 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:54 smithi066 ceph-mon[25664]: [21/Mar/2024:21:33:54] ENGINE Bus STARTING 2024-03-21T21:33:55.477 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:55.303+0000 7fe22687c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:33:55.477 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:55.399+0000 7fe22687c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:33:55.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:55.547+0000 7fe22687c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:33:55.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:55.647+0000 7fe22687c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:33:56.041 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-mon[25664]: [21/Mar/2024:21:33:54] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:33:56.041 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-mon[25664]: [21/Mar/2024:21:33:54] ENGINE Serving on http://172.21.15.28:8765 2024-03-21T21:33:56.041 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-mon[25664]: [21/Mar/2024:21:33:54] ENGINE Bus STARTED 2024-03-21T21:33:56.041 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-mon[25664]: mgrmap e28: y(active, since 1.74672s) 2024-03-21T21:33:56.041 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-mon[25664]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:56.041 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:55.753+0000 7fe22687c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:33:56.041 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:56 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:56.041+0000 7fe22687c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T21:33:56.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[29568]: [21/Mar/2024:21:33:54] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:33:56.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[29568]: [21/Mar/2024:21:33:54] ENGINE Serving on http://172.21.15.28:8765 2024-03-21T21:33:56.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[29568]: [21/Mar/2024:21:33:54] ENGINE Bus STARTED 2024-03-21T21:33:56.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[29568]: mgrmap e28: y(active, since 1.74672s) 2024-03-21T21:33:56.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[29568]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:56.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[24806]: [21/Mar/2024:21:33:54] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:33:56.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[24806]: [21/Mar/2024:21:33:54] ENGINE Serving on http://172.21.15.28:8765 2024-03-21T21:33:56.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[24806]: [21/Mar/2024:21:33:54] ENGINE Bus STARTED 2024-03-21T21:33:56.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[24806]: mgrmap e28: y(active, since 1.74672s) 2024-03-21T21:33:56.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:55 smithi028 ceph-mon[24806]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:56.347 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:56 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:56.157+0000 7fe22687c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:33:56.347 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:56 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:56.247+0000 7fe22687c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:33:56.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:56 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:56.347+0000 7fe22687c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:33:56.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:56 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:56.544+0000 7fe22687c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:33:57.020 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:56 smithi028 ceph-mon[29568]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:33:57.020 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:56 smithi028 ceph-mon[29568]: Cluster is now healthy 2024-03-21T21:33:57.020 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:56 smithi028 ceph-mon[29568]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:57.020 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:56 smithi028 ceph-mon[29568]: mgrmap e29: y(active, since 2s) 2024-03-21T21:33:57.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:56 smithi028 ceph-mon[24806]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:33:57.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:56 smithi028 ceph-mon[24806]: Cluster is now healthy 2024-03-21T21:33:57.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:56 smithi028 ceph-mon[24806]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:57.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:56 smithi028 ceph-mon[24806]: mgrmap e29: y(active, since 2s) 2024-03-21T21:33:57.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:56 smithi066 ceph-mon[25664]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:33:57.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:56 smithi066 ceph-mon[25664]: Cluster is now healthy 2024-03-21T21:33:57.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:56 smithi066 ceph-mon[25664]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:57.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:56 smithi066 ceph-mon[25664]: mgrmap e29: y(active, since 2s) 2024-03-21T21:33:57.782 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:57.632+0000 7fe22687c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:33:57.783 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:57.782+0000 7fe22687c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:33:58.154 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:58.153+0000 7fe22687c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:33:58.494 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:58.261+0000 7fe22687c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:33:58.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:58.495+0000 7fe22687c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:33:58.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:58.701+0000 7fe22687c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:33:59.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:58 smithi028 ceph-mon[29568]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:59.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:58 smithi028 ceph-mon[29568]: mgrmap e30: y(active, since 4s) 2024-03-21T21:33:59.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:58 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:33:59.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:33:58 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:33:59.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:58 smithi028 ceph-mon[24806]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:59.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:58 smithi028 ceph-mon[24806]: mgrmap e30: y(active, since 4s) 2024-03-21T21:33:59.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:58 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:33:59.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:33:58 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:33:59.249 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:59 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:59.016+0000 7fe22687c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:33:59.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:58 smithi066 ceph-mon[25664]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:33:59.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:58 smithi066 ceph-mon[25664]: mgrmap e30: y(active, since 4s) 2024-03-21T21:33:59.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:58 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:33:59.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:33:58 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:33:59.554 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:59 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:59.297+0000 7fe22687c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:34:00.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:33:59 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:33:59.555+0000 7fe22687c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:34:00.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:00 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:34:00.418+0000 7fe22687c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:34:00.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:00 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:00 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:00 smithi066 ceph-mon[25664]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:34:00.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:00 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:00 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[24806]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:34:00.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[29568]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:34:00.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:00.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:00 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:01.361 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:34:01.022+0000 7fe22687c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:34:01.361 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:34:01.126+0000 7fe22687c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:34:01.361 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:34:01.361+0000 7fe22687c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:34:01.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:34:01] ENGINE Bus STARTING 2024-03-21T21:34:01.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: CherryPy Checker: 2024-03-21T21:34:01.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: The Application mounted at '' has an empty config. 2024-03-21T21:34:01.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:34:01.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:34:01] ENGINE Serving on http://:::9283 2024-03-21T21:34:01.749 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:34:01] ENGINE Bus STARTED 2024-03-21T21:34:02.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: Standby manager daemon x started 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:34:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: Standby manager daemon x started 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:34:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:34:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:34:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:34:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:02 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:34:02.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: Standby manager daemon x started 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:02.999 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:34:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:34:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:34:03.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:34:03.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:34:03.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:34:03.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:34:03.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:34:03.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:34:03.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: mgrmap e31: y(active, since 9s), standbys: x 2024-03-21T21:34:03.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:03.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:03.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-21T21:34:03.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:03.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:34:03.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:34:03.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:34:03.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:34:03.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:03 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:34:03.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: mgrmap e31: y(active, since 9s), standbys: x 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:34:03.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: mgrmap e31: y(active, since 9s), standbys: x 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:34:03.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:34:03.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:34:03.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:34:03.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:03 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:34:04.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:04 smithi066 ceph-mon[25664]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:34:04.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:04 smithi028 ceph-mon[29568]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:34:04.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:04 smithi028 ceph-mon[24806]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:34:05.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:05 smithi028 ceph-mon[24806]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:05.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:05 smithi028 ceph-mon[24806]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:05.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:05 smithi028 ceph-mon[29568]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:05.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:05 smithi028 ceph-mon[29568]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:05.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:05 smithi066 ceph-mon[25664]: from='client.15114 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:05.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:05 smithi066 ceph-mon[25664]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:06.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:05.751Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-21T21:34:06.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:05.752Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-21T21:34:06.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:05.753Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-21T21:34:06.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:05.758Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-21T21:34:06.249 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:05.758Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-21T21:34:07.736 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:07 smithi066 ceph-mon[25664]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:07.736 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:07 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:07.736 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:07 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:07.736 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:07 smithi066 ceph-mon[25664]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:34:07.736 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:07 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/59092466' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:34:07.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[29568]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:07.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:07.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:07.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[29568]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:34:07.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/59092466' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:34:07.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[24806]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:07.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:07.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:07.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[24806]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:34:07.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:07 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/59092466' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:34:08.823 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:08 smithi066 ceph-mon[25664]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:34:08.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:08 smithi028 ceph-mon[29568]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:34:08.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:08 smithi028 ceph-mon[24806]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:34:09.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:09 smithi028 ceph-mon[24806]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:09.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:09 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:09 smithi028 ceph-mon[29568]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:09 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:09.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:09 smithi066 ceph-mon[25664]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:09.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:09 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:10.377 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 systemd[1]: Stopping Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:34:10.377 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 bash[55027]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:34:10.678 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.377Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:34:10.678 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.377Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T21:34:10.678 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.377Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T21:34:10.678 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.377Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T21:34:10.678 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.377Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T21:34:10.679 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.377Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T21:34:10.679 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.378Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T21:34:10.679 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.378Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T21:34:10.679 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.378Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T21:34:10.679 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.378Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T21:34:10.679 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.378Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T21:34:10.679 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[49248]: ts=2024-03-21T21:34:10.378Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T21:34:10.679 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 podman[55040]: 2024-03-21 21:34:10.387696685 +0000 UTC m=+0.132944189 container died cc8b8c4750568d403064206ce42731b0b2020bb9b0d7e2767afb16b1ccfde2bc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:34:10.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 podman[55040]: 2024-03-21 21:34:10.75177834 +0000 UTC m=+0.497025846 container cleanup cc8b8c4750568d403064206ce42731b0b2020bb9b0d7e2767afb16b1ccfde2bc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:34:10.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 bash[55040]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T21:34:10.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:10 smithi066 bash[55067]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:34:11.278 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 podman[55054]: 2024-03-21 21:34:11.027853391 +0000 UTC m=+0.642086296 container remove cc8b8c4750568d403064206ce42731b0b2020bb9b0d7e2767afb16b1ccfde2bc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:34:11.278 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T21:34:11.278 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 systemd[1]: Stopped Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:34:11.278 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 2.067s CPU time. 2024-03-21T21:34:11.278 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 systemd[1]: Starting Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:34:11.278 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 podman[55137]: 2024-03-21 21:34:11.179412632 +0000 UTC m=+0.020780534 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T21:34:11.278 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 podman[55137]: 2024-03-21 21:34:11.278744792 +0000 UTC m=+0.120112694 container create 77364ca6b9a3195d75400b11b226d78c6ec411ce7c7bee0ecb0a73cf360a0e48 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:34:11.549 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 podman[55137]: 2024-03-21 21:34:11.522703448 +0000 UTC m=+0.364071348 container init 77364ca6b9a3195d75400b11b226d78c6ec411ce7c7bee0ecb0a73cf360a0e48 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:34:11.550 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 podman[55137]: 2024-03-21 21:34:11.525955264 +0000 UTC m=+0.367323164 container start 77364ca6b9a3195d75400b11b226d78c6ec411ce7c7bee0ecb0a73cf360a0e48 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:34:11.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:11 smithi028 ceph-mon[29568]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:11.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:11 smithi028 ceph-mon[24806]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:11.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-mon[25664]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:34:11.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.550Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T21:34:11.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.550Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T21:34:11.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.550Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T21:34:11.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.550Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi066 (none))" 2024-03-21T21:34:11.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.550Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T21:34:11.999 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.550Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T21:34:12.000 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.553Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T21:34:12.000 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.553Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T21:34:12.000 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.555Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T21:34:12.000 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.557Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T21:34:12.000 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.557Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.591µs 2024-03-21T21:34:12.000 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.557Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T21:34:12.000 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 bash[55137]: 77364ca6b9a3195d75400b11b226d78c6ec411ce7c7bee0ecb0a73cf360a0e48 2024-03-21T21:34:12.000 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 systemd[1]: Started Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:34:12.000 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:11.780Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-03-21T21:34:12.296 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:12.043Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-03-21T21:34:12.296 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:12.266Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-03-21T21:34:12.296 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:12.266Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-03-21T21:34:12.296 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:12.266Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=37.68µs wal_replay_duration=709.609601ms total_replay_duration=709.664432ms 2024-03-21T21:34:12.296 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:12.271Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T21:34:12.296 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:12.271Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T21:34:12.297 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:12.271Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T21:34:12.297 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:12.296Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.82834ms db_storage=846ns remote_storage=2.019µs web_handler=387ns query_engine=731ns scrape=627.386µs scrape_sd=154.03µs notify=21.083µs notify_sd=16.461µs rules=23.478735ms 2024-03-21T21:34:12.297 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:34:12.296Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: Adding iSCSI gateway http://:@172.21.15.28:5000 to Dashboard 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:34:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[29568]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 1 op/s 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: Adding iSCSI gateway http://:@172.21.15.28:5000 to Dashboard 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:34:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:34:12.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:34:12.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:34:12.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:34:12.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:12 smithi028 ceph-mon[24806]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 1 op/s 2024-03-21T21:34:12.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:34:12.995 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: Adding iSCSI gateway http://:@172.21.15.28:5000 to Dashboard 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi028"}]: dispatch 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:34:12.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:12 smithi066 ceph-mon[25664]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 1 op/s 2024-03-21T21:34:15.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:15 smithi028 ceph-mon[29568]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:34:15.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:15 smithi028 ceph-mon[24806]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:34:15.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:15 smithi066 ceph-mon[25664]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:34:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:34:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:34:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:34:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:34:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:16 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:16 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:16 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:16 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:16 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:16 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:34:17.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:16 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:34:17.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:16 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:34:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:17 smithi028 ceph-mon[29568]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:17 smithi028 ceph-mon[29568]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 1 op/s 2024-03-21T21:34:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:17 smithi028 ceph-mon[24806]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:17 smithi028 ceph-mon[24806]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 1 op/s 2024-03-21T21:34:17.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:17 smithi066 ceph-mon[25664]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:17.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:17 smithi066 ceph-mon[25664]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 1 op/s 2024-03-21T21:34:18.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:18 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:18.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:18 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:18.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:18 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:19.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:19 smithi028 ceph-mon[29568]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 1 op/s 2024-03-21T21:34:19.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:19 smithi028 ceph-mon[24806]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 1 op/s 2024-03-21T21:34:19.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:19 smithi066 ceph-mon[25664]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 1 op/s 2024-03-21T21:34:21.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:34:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:34:20] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-03-21T21:34:21.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:21 smithi028 ceph-mon[24806]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:34:21.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:21 smithi028 ceph-mon[29568]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:34:21.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:21 smithi066 ceph-mon[25664]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:34:23.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:23 smithi066 ceph-mon[25664]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:34:23.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:23 smithi028 ceph-mon[24806]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:34:23.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:23 smithi028 ceph-mon[29568]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:34:24.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:24 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:24.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:24 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:24.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:24 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:25.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:25 smithi028 ceph-mon[29568]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:34:25.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:25 smithi028 ceph-mon[24806]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:34:25.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:25 smithi066 ceph-mon[25664]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:34:27.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:27 smithi028 ceph-mon[29568]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:34:27.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:27 smithi028 ceph-mon[24806]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:34:27.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:27 smithi066 ceph-mon[25664]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:34:28.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:28 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:28.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:28 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:28.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:28 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:29.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:29 smithi028 ceph-mon[24806]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:29.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:29 smithi028 ceph-mon[29568]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:29.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:29 smithi066 ceph-mon[25664]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:31.112 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:34:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:34:30] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-21T21:34:31.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:31 smithi028 ceph-mon[24806]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:31.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:31 smithi028 ceph-mon[29568]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:31.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:31 smithi066 ceph-mon[25664]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:33 smithi028 ceph-mon[29568]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:33.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:33 smithi028 ceph-mon[24806]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:33.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:33 smithi066 ceph-mon[25664]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:34.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:34 smithi028 ceph-mon[29568]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:34.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:34 smithi028 ceph-mon[24806]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:34.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:34 smithi066 ceph-mon[25664]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:37.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:37 smithi028 ceph-mon[29568]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:37.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:37 smithi028 ceph-mon[24806]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:37.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:37 smithi066 ceph-mon[25664]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:38.493 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:38 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:38.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:38 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:38 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:39.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:39 smithi028 ceph-mon[24806]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:39.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:39 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:39.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:39 smithi028 ceph-mon[29568]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:39.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:39 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:39.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:39 smithi066 ceph-mon[25664]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:39.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:39 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:41.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:34:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:34:40] "GET /metrics HTTP/1.1" 200 37971 "" "Prometheus/2.33.4" 2024-03-21T21:34:41.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:41 smithi028 ceph-mon[24806]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:41.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:41 smithi028 ceph-mon[29568]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:41.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:41 smithi066 ceph-mon[25664]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:43 smithi028 ceph-mon[24806]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:43.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:43 smithi028 ceph-mon[29568]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:43.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:43 smithi066 ceph-mon[25664]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:45.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:45 smithi028 ceph-mon[24806]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:45.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:45 smithi028 ceph-mon[29568]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:45.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:45 smithi066 ceph-mon[25664]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:47.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:47 smithi028 ceph-mon[24806]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:47.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:47 smithi028 ceph-mon[29568]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:47.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:47 smithi066 ceph-mon[25664]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:48.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:48 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:48.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:48 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:48.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:48 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:49.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:49 smithi028 ceph-mon[29568]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:49.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:49 smithi028 ceph-mon[24806]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:49.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:49 smithi066 ceph-mon[25664]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:51.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:34:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:34:50] "GET /metrics HTTP/1.1" 200 37971 "" "Prometheus/2.33.4" 2024-03-21T21:34:51.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:51 smithi028 ceph-mon[29568]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:51.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:51 smithi028 ceph-mon[24806]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:51.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:51 smithi066 ceph-mon[25664]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:53.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:53 smithi028 ceph-mon[29568]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:53.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:53 smithi028 ceph-mon[24806]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:53.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:53 smithi066 ceph-mon[25664]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:54.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:54.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:54.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:34:55.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:55 smithi028 ceph-mon[24806]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:55.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:55 smithi028 ceph-mon[29568]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:55.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:55 smithi066 ceph-mon[25664]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:57.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:57 smithi028 ceph-mon[24806]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:57.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:57 smithi028 ceph-mon[29568]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:57.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:57 smithi066 ceph-mon[25664]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:34:58.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:58 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:58.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:58 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:58.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:58 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:34:59.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:34:59 smithi028 ceph-mon[29568]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:59.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:34:59 smithi028 ceph-mon[24806]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:34:59.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:34:59 smithi066 ceph-mon[25664]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:01.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:35:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:35:00] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-03-21T21:35:01.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:01 smithi028 ceph-mon[24806]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:01.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:01 smithi028 ceph-mon[29568]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:01.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:01 smithi066 ceph-mon[25664]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:03.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:03 smithi066 ceph-mon[25664]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:03.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:03 smithi028 ceph-mon[29568]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:03.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:03 smithi028 ceph-mon[24806]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:05.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:05 smithi066 ceph-mon[25664]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:05.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:05 smithi028 ceph-mon[24806]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:05.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:05 smithi028 ceph-mon[29568]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:07.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:07 smithi066 ceph-mon[25664]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:07.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:07 smithi028 ceph-mon[29568]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:07.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:07 smithi028 ceph-mon[24806]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:08.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:08 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:08.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:08 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:08.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:08 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:09.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:09 smithi066 ceph-mon[25664]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:09.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:09 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:09.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:09 smithi028 ceph-mon[24806]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:09.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:09 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:09 smithi028 ceph-mon[29568]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:09 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:11.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:35:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:35:10] "GET /metrics HTTP/1.1" 200 37967 "" "Prometheus/2.33.4" 2024-03-21T21:35:11.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:11 smithi066 ceph-mon[25664]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:11.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:11 smithi028 ceph-mon[24806]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:11.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:11 smithi028 ceph-mon[29568]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:13.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:13 smithi066 ceph-mon[25664]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:13.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:13 smithi028 ceph-mon[29568]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:13.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:13 smithi028 ceph-mon[24806]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:15.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:15 smithi066 ceph-mon[25664]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:15.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:15 smithi028 ceph-mon[24806]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:15 smithi028 ceph-mon[29568]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:16.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:16 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:35:16.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:16 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:35:16.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:16 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:35:17.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:17 smithi066 ceph-mon[25664]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:17.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:17 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:17 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:17 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:17 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:17 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:35:17.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:17 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:35:17.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:17 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[24806]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:35:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:35:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[29568]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:35:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:35:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:17 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:35:18.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:18 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:18.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:18 smithi066 ceph-mon[25664]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 906 B/s rd, 0 op/s 2024-03-21T21:35:18.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:18 smithi066 ceph-mon[25664]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:18.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:18 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:18 smithi028 ceph-mon[29568]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 906 B/s rd, 0 op/s 2024-03-21T21:35:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:18 smithi028 ceph-mon[29568]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:18.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:18 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:18.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:18 smithi028 ceph-mon[24806]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 906 B/s rd, 0 op/s 2024-03-21T21:35:18.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:18 smithi028 ceph-mon[24806]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:20.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:20 smithi066 ceph-mon[25664]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 551 B/s rd, 0 op/s 2024-03-21T21:35:20.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:20 smithi028 ceph-mon[29568]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 551 B/s rd, 0 op/s 2024-03-21T21:35:20.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:20 smithi028 ceph-mon[24806]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 551 B/s rd, 0 op/s 2024-03-21T21:35:21.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:35:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:35:20] "GET /metrics HTTP/1.1" 200 37967 "" "Prometheus/2.33.4" 2024-03-21T21:35:22.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:22 smithi066 ceph-mon[25664]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:22.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:22 smithi028 ceph-mon[29568]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:23.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:22 smithi028 ceph-mon[24806]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:24.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:24 smithi028 ceph-mon[29568]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:24.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:24 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:24.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:24 smithi028 ceph-mon[24806]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:24.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:24 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:24.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:24 smithi066 ceph-mon[25664]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:24.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:24 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:26 smithi066 ceph-mon[25664]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:26.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:26 smithi028 ceph-mon[24806]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:26.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:26 smithi028 ceph-mon[29568]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:35:28.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:28 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:28.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:28 smithi028 ceph-mon[24806]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:35:28.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:28 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:28.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:28 smithi028 ceph-mon[29568]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:35:28.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:28 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:28.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:28 smithi066 ceph-mon[25664]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:35:30.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:30 smithi028 ceph-mon[29568]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:30.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:30 smithi028 ceph-mon[24806]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:30.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:30 smithi066 ceph-mon[25664]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:31.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:35:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:35:30] "GET /metrics HTTP/1.1" 200 37969 "" "Prometheus/2.33.4" 2024-03-21T21:35:32.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:32 smithi028 ceph-mon[24806]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:32.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:32 smithi028 ceph-mon[29568]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:32.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:32 smithi066 ceph-mon[25664]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:34.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:34 smithi028 ceph-mon[24806]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:34.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:34 smithi028 ceph-mon[29568]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:34.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:34 smithi066 ceph-mon[25664]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:36.768 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:36 smithi028 ceph-mon[24806]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:36.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:36 smithi028 ceph-mon[29568]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:36.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:36 smithi066 ceph-mon[25664]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:38.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:38 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:38.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:38 smithi028 ceph-mon[29568]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:38 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:38 smithi028 ceph-mon[24806]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:38.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:38 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:38.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:38 smithi066 ceph-mon[25664]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:39.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:39 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:39.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:39 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:39.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:39 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:40.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:40 smithi028 ceph-mon[29568]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:40.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:40 smithi028 ceph-mon[24806]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:40.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:40 smithi066 ceph-mon[25664]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:41.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:35:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:35:40] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-03-21T21:35:42.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:42 smithi028 ceph-mon[24806]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:42.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:42 smithi028 ceph-mon[29568]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:42.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:42 smithi066 ceph-mon[25664]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:44.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:44 smithi028 ceph-mon[24806]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:44.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:44 smithi028 ceph-mon[29568]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:44.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:44 smithi066 ceph-mon[25664]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:46.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:46 smithi028 ceph-mon[24806]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:46.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:46 smithi028 ceph-mon[29568]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:46.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:46 smithi066 ceph-mon[25664]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:48.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:48 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:48.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:48 smithi028 ceph-mon[29568]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:48.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:48 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:48.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:48 smithi028 ceph-mon[24806]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:48.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:48 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:48.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:48 smithi066 ceph-mon[25664]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:50.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:50 smithi028 ceph-mon[29568]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:50.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:50 smithi028 ceph-mon[24806]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:50.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:50 smithi066 ceph-mon[25664]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:51.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:35:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:35:50] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-03-21T21:35:52.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:52 smithi028 ceph-mon[24806]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:52.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:52 smithi028 ceph-mon[29568]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:52 smithi066 ceph-mon[25664]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:54.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:54 smithi028 ceph-mon[24806]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:54.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:54.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:54 smithi028 ceph-mon[29568]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:54.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:54.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:54 smithi066 ceph-mon[25664]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:54.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:35:56.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:56 smithi028 ceph-mon[24806]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:56.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:56 smithi028 ceph-mon[29568]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:56.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:56 smithi066 ceph-mon[25664]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:35:58.803 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:58 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:58.803 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:35:58 smithi066 ceph-mon[25664]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:58.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:58 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:58.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:35:58 smithi028 ceph-mon[29568]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:35:58.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:58 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:35:58.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:35:58 smithi028 ceph-mon[24806]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:00.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:00 smithi028 ceph-mon[29568]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:00.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:00 smithi028 ceph-mon[24806]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:00.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:00 smithi066 ceph-mon[25664]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:01.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:36:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:36:00] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-21T21:36:02.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:02 smithi028 ceph-mon[29568]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:02.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:02 smithi028 ceph-mon[24806]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:02.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:02 smithi066 ceph-mon[25664]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:04.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:04 smithi028 ceph-mon[29568]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:04.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:04 smithi028 ceph-mon[24806]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:04.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:04 smithi066 ceph-mon[25664]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:06.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:06 smithi028 ceph-mon[29568]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:06.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:06 smithi028 ceph-mon[24806]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:06.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:06 smithi066 ceph-mon[25664]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:08.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:08 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:08.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:08 smithi028 ceph-mon[24806]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:08.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:08 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:08.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:08 smithi028 ceph-mon[29568]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:08.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:08 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:08.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:08 smithi066 ceph-mon[25664]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:09.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:09 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:09 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:09.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:09 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:10.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:10 smithi028 ceph-mon[24806]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:10.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:10 smithi028 ceph-mon[29568]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:10.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:10 smithi066 ceph-mon[25664]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:11.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:36:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:36:10] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-21T21:36:12.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:12 smithi028 ceph-mon[24806]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:12.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:12 smithi028 ceph-mon[29568]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:12.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:12 smithi066 ceph-mon[25664]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:14.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:14 smithi028 ceph-mon[24806]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:14.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:14 smithi028 ceph-mon[29568]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:14.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:14 smithi066 ceph-mon[25664]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:16.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:16 smithi028 ceph-mon[24806]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:16.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:16 smithi028 ceph-mon[29568]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:16 smithi066 ceph-mon[25664]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[29568]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:36:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:36:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:36:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[29568]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:36:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[24806]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:36:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:36:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:36:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:18 smithi028 ceph-mon[24806]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:36:18.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:18 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:18.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:18 smithi066 ceph-mon[25664]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:18.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:18 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:36:18.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:18 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:36:19.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:18 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:36:19.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:18 smithi066 ceph-mon[25664]: from='mgr.25087 ' entity='mgr.y' 2024-03-21T21:36:19.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:19 smithi028 ceph-mon[29568]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T21:36:19.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:19 smithi028 ceph-mon[29568]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:19.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:19 smithi028 ceph-mon[24806]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T21:36:19.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:19 smithi028 ceph-mon[24806]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:19.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:19 smithi066 ceph-mon[25664]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T21:36:19.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:19 smithi066 ceph-mon[25664]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:21.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:36:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:36:20] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-21T21:36:21.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:21 smithi028 ceph-mon[24806]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:21.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:21 smithi028 ceph-mon[29568]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:21.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:21 smithi066 ceph-mon[25664]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:23.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:23 smithi028 ceph-mon[24806]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:23.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:23 smithi028 ceph-mon[29568]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:23.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:23 smithi066 ceph-mon[25664]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:24.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:24 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:24.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:24 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:24.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:24 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:25.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:25 smithi028 ceph-mon[24806]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T21:36:25.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:25 smithi028 ceph-mon[29568]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T21:36:25.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:25 smithi066 ceph-mon[25664]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T21:36:27.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:27 smithi028 ceph-mon[24806]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:27.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:27 smithi028 ceph-mon[29568]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:27.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:27 smithi066 ceph-mon[25664]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:36:28.901 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:28 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:28.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:28 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:28.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:28 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:29.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:29 smithi028 ceph-mon[29568]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:36:29.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:29 smithi028 ceph-mon[24806]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:36:29.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:29 smithi066 ceph-mon[25664]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:36:31.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:36:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:36:30] "GET /metrics HTTP/1.1" 200 37971 "" "Prometheus/2.33.4" 2024-03-21T21:36:31.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:31 smithi028 ceph-mon[29568]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:31.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:31 smithi028 ceph-mon[24806]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:31.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:31 smithi066 ceph-mon[25664]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:33.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:33 smithi028 ceph-mon[24806]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:33.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:33 smithi028 ceph-mon[29568]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:33.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:33 smithi066 ceph-mon[25664]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:35.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:35 smithi028 ceph-mon[29568]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:35.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:35 smithi028 ceph-mon[24806]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:35.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:35 smithi066 ceph-mon[25664]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:37 smithi028 ceph-mon[29568]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:37 smithi028 ceph-mon[24806]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:37.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:37 smithi066 ceph-mon[25664]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:38.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:38 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:38 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:38.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:38 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:39.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:39 smithi028 ceph-mon[24806]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:39.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:39 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:39.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:39 smithi028 ceph-mon[29568]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:39.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:39 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:39.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:39 smithi066 ceph-mon[25664]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:39.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:39 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:41.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:36:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:36:40] "GET /metrics HTTP/1.1" 200 37978 "" "Prometheus/2.33.4" 2024-03-21T21:36:41.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:41 smithi028 ceph-mon[24806]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:41.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:41 smithi028 ceph-mon[29568]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:41.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:41 smithi066 ceph-mon[25664]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:43.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:43 smithi028 ceph-mon[29568]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:43.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:43 smithi028 ceph-mon[24806]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:43.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:43 smithi066 ceph-mon[25664]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:45.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:45 smithi028 ceph-mon[29568]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:45.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:45 smithi028 ceph-mon[24806]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:45.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:45 smithi066 ceph-mon[25664]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:47.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:47 smithi028 ceph-mon[29568]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:47.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:47 smithi028 ceph-mon[24806]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:47.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:47 smithi066 ceph-mon[25664]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:48.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:48 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:48.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:48 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:48.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:48 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:49.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:49 smithi028 ceph-mon[29568]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:36:49.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:49 smithi028 ceph-mon[24806]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:36:49.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:49 smithi066 ceph-mon[25664]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:36:51.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:36:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:36:50] "GET /metrics HTTP/1.1" 200 37978 "" "Prometheus/2.33.4" 2024-03-21T21:36:51.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:51 smithi028 ceph-mon[29568]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:51.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:51 smithi028 ceph-mon[24806]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:51.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:51 smithi066 ceph-mon[25664]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:53.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:53 smithi028 ceph-mon[29568]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:36:53.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:53 smithi028 ceph-mon[24806]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:36:53.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:53 smithi066 ceph-mon[25664]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:36:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:54 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:54.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:54 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:54.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:54 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:36:55.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:55 smithi028 ceph-mon[29568]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:36:55.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:55 smithi028 ceph-mon[24806]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:36:55.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:55 smithi066 ceph-mon[25664]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:36:57.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:57 smithi028 ceph-mon[29568]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:57.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:57 smithi028 ceph-mon[24806]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:57.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:57 smithi066 ceph-mon[25664]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:36:58.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:36:58.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:58 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:58.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:58 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:58.901 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:58 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:36:59.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:36:59 smithi028 ceph-mon[24806]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:59.740 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:36:59 smithi028 ceph-mon[29568]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:36:59.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:36:59 smithi066 ceph-mon[25664]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:00.188 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:37:00.188 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (9m) 2m ago 17m 23.3M - 0.23.0 ba2b418f427c cd065b97fad1 2024-03-21T21:37:00.188 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 error 2m ago 16m - - 2024-03-21T21:37:00.188 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (2m) 2m ago 16m 46.0M - 3.5 e1d6a67b021e 9ecf3caf88b1 2024-03-21T21:37:00.188 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283 running (11m) 2m ago 21m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 7fe31a1b09b6 2024-03-21T21:37:00.188 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (6m) 2m ago 23m 500M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:37:00.188 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (23m) 2m ago 23m 69.9M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:37:00.188 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (22m) 2m ago 22m 50.5M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:37:00.188 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (22m) 2m ago 22m 49.9M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (9m) 2m ago 17m 17.4M - 1.3.1 1dbe0e931976 a8a424c65ebe 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (8m) 2m ago 17m 19.4M - 1.3.1 1dbe0e931976 f6cdb9cb9923 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (21m) 2m ago 21m 52.8M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (21m) 2m ago 21m 53.6M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (20m) 2m ago 20m 52.1M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (20m) 2m ago 20m 51.6M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (19m) 2m ago 19m 52.2M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (19m) 2m ago 19m 51.4M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (19m) 2m ago 19m 51.1M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (18m) 2m ago 18m 51.8M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (2m) 2m ago 17m 44.8M - 2.33.4 514e6a882f6e 77364ca6b9a3 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (16m) 2m ago 16m 75.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:37:00.189 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (16m) 2m ago 16m 73.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:37:00.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions' 2024-03-21T21:37:01.118 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:37:00] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-21T21:37:01.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:01 smithi028 ceph-mon[29568]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:37:01.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:01 smithi028 ceph-mon[29568]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:01 smithi028 ceph-mon[24806]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:37:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:01 smithi028 ceph-mon[24806]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:01.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:01 smithi066 ceph-mon[25664]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:37:02.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:01 smithi066 ceph-mon[25664]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:03.065 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:37:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:37:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:37:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:37:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:37:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:37:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:37:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:37:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:37:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:37:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:37:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:37:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:37:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:37:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:37:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T21:37:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:37:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:37:03.068 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:37:03.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:03 smithi028 ceph-mon[29568]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:03.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:03 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1183226506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:37:03.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:03 smithi028 ceph-mon[24806]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:03.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:03 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1183226506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:37:03.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph -s' 2024-03-21T21:37:03.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:03 smithi066 ceph-mon[25664]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:04.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:03 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1183226506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:37:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: cluster: 2024-03-21T21:37:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: id: c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:37:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: health: HEALTH_WARN 2024-03-21T21:37:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: 1 failed cephadm daemon(s) 2024-03-21T21:37:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:37:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: services: 2024-03-21T21:37:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: mon: 3 daemons, quorum a,c,b (age 22m) 2024-03-21T21:37:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: mgr: y(active, since 3m), standbys: x 2024-03-21T21:37:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: data: 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: pools: 6 pools, 161 pgs 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: objects: 209 objects, 585 KiB 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: usage: 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: pgs: 161 active+clean 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: io: 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-21T21:37:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T21:37:05.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:05 smithi028 ceph-mon[29568]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:05.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:05 smithi028 ceph-mon[24806]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:05.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:05 smithi066 ceph-mon[25664]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:06.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:37:06.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:06 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2577807451' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:37:06.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:06 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2577807451' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:37:06.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:06 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2577807451' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T21:37:07.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:07 smithi028 ceph-mon[29568]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:07.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:07 smithi028 ceph-mon[24806]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:07.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:07 smithi066 ceph-mon[25664]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:08.725 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T21:37:08.725 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T21:37:08.725 INFO:teuthology.orchestra.run.smithi028.stdout: daemon grafana.a on smithi066 is in error state 2024-03-21T21:37:08.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:08 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:37:08.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:08 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:37:08.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:08 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:37:09.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-21T21:37:09.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:09 smithi028 ceph-mon[24806]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:09.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:09 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/4218000834' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:37:09.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:09 smithi028 ceph-mon[24806]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:09.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:09 smithi028 ceph-mon[29568]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:09.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:09 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/4218000834' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:37:09.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:09 smithi028 ceph-mon[29568]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:09.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:09 smithi066 ceph-mon[25664]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:10.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:09 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/4218000834' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:37:10.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:09 smithi066 ceph-mon[25664]: from='mgr.25087 172.21.15.28:0/1988143578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:11.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:37:10] "GET /metrics HTTP/1.1" 200 37970 "" "Prometheus/2.33.4" 2024-03-21T21:37:11.681 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:37:11.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:11 smithi028 ceph-mon[24806]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:11.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:11 smithi028 ceph-mon[29568]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:11.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:11 smithi066 ceph-mon[25664]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:12.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph mgr fail' 2024-03-21T21:37:12.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:12 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2054174878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:37:12.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:12 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2054174878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:37:12.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:12 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2054174878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:37:13.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:13 smithi028 ceph-mon[24806]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:13.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:13 smithi028 ceph-mon[29568]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:13.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:13 smithi066 ceph-mon[25664]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:14.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-mon[25664]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:14.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1481212119' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:37:14.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-mon[25664]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:37:14.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-mon[25664]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T21:37:14.909 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:37:14] ENGINE Bus STOPPING 2024-03-21T21:37:14.909 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:37:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:37:14.909 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:37:14] ENGINE Bus STOPPED 2024-03-21T21:37:14.909 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:37:14] ENGINE Bus STARTING 2024-03-21T21:37:14.910 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:37:14] ENGINE Serving on http://:::9283 2024-03-21T21:37:14.910 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:37:14] ENGINE Bus STARTED 2024-03-21T21:37:14.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-mon[29568]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:14.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1481212119' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:37:14.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:37:14.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-mon[29568]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T21:37:14.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-mon[24806]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:37:14.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1481212119' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:37:14.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-mon[24806]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T21:37:14.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-mon[24806]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T21:37:14.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:14.644+0000 7fbfee920700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-21T21:37:14.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ignoring --setuser ceph since I am not root 2024-03-21T21:37:14.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ignoring --setgroup ceph since I am not root 2024-03-21T21:37:14.933 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:14 smithi028 ceph-mgr[62478]: -- 172.21.15.28:0/1742903142 <== mon.1 v2:172.21.15.28:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5612157d65a0 con 0x5612157ab000 2024-03-21T21:37:15.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'sleep 180' 2024-03-21T21:37:15.641 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:15.244+0000 7f8a93d92200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:37:15.641 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:15.348+0000 7f8a93d92200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:37:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T21:37:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: mgrmap e32: x(active, starting, since 0.280104s) 2024-03-21T21:37:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:37:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: Manager daemon x is now available 2024-03-21T21:37:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: mgrmap e32: x(active, starting, since 0.280104s) 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:37:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: Manager daemon x is now available 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:37:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:15 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: mgrmap e32: x(active, starting, since 0.280104s) 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: Manager daemon x is now available 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:37:15.977 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:15 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T21:37:16.297 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:16.050+0000 7f8a93d92200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:37:16.298 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:16.146+0000 7f8a93d92200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:37:16.650 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:16.296+0000 7f8a93d92200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:37:16.651 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:16.397+0000 7f8a93d92200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:37:16.651 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:16.484+0000 7f8a93d92200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:37:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[29568]: [21/Mar/2024:21:37:15] ENGINE Bus STARTING 2024-03-21T21:37:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[29568]: [21/Mar/2024:21:37:15] ENGINE Serving on https://172.21.15.66:7150 2024-03-21T21:37:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[29568]: mgrmap e33: x(active, since 1.28615s) 2024-03-21T21:37:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[29568]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[29568]: [21/Mar/2024:21:37:15] ENGINE Serving on http://172.21.15.66:8765 2024-03-21T21:37:16.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[29568]: [21/Mar/2024:21:37:15] ENGINE Bus STARTED 2024-03-21T21:37:16.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:16.770+0000 7f8a93d92200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T21:37:16.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:16.886+0000 7f8a93d92200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:37:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[24806]: [21/Mar/2024:21:37:15] ENGINE Bus STARTING 2024-03-21T21:37:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[24806]: [21/Mar/2024:21:37:15] ENGINE Serving on https://172.21.15.66:7150 2024-03-21T21:37:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[24806]: mgrmap e33: x(active, since 1.28615s) 2024-03-21T21:37:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[24806]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[24806]: [21/Mar/2024:21:37:15] ENGINE Serving on http://172.21.15.66:8765 2024-03-21T21:37:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-mon[24806]: [21/Mar/2024:21:37:15] ENGINE Bus STARTED 2024-03-21T21:37:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:16 smithi066 ceph-mon[25664]: [21/Mar/2024:21:37:15] ENGINE Bus STARTING 2024-03-21T21:37:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:16 smithi066 ceph-mon[25664]: [21/Mar/2024:21:37:15] ENGINE Serving on https://172.21.15.66:7150 2024-03-21T21:37:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:16 smithi066 ceph-mon[25664]: mgrmap e33: x(active, since 1.28615s) 2024-03-21T21:37:16.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:16 smithi066 ceph-mon[25664]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:17.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:16 smithi066 ceph-mon[25664]: [21/Mar/2024:21:37:15] ENGINE Serving on http://172.21.15.66:8765 2024-03-21T21:37:17.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:16 smithi066 ceph-mon[25664]: [21/Mar/2024:21:37:15] ENGINE Bus STARTED 2024-03-21T21:37:17.234 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:16.978+0000 7f8a93d92200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:37:17.235 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:17.081+0000 7f8a93d92200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:37:17.574 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:17.263+0000 7f8a93d92200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:37:17.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:17 smithi066 ceph-mon[25664]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:37:17.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:17 smithi066 ceph-mon[25664]: Cluster is now healthy 2024-03-21T21:37:17.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:17 smithi066 ceph-mon[25664]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:17.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:17 smithi066 ceph-mon[25664]: mgrmap e34: x(active, since 2s) 2024-03-21T21:37:17.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:17 smithi066 ceph-mon[25664]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:37:17.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[29568]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:37:17.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[29568]: Cluster is now healthy 2024-03-21T21:37:17.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[29568]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:17.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[29568]: mgrmap e34: x(active, since 2s) 2024-03-21T21:37:17.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[29568]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:37:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[24806]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:37:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[24806]: Cluster is now healthy 2024-03-21T21:37:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[24806]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[24806]: mgrmap e34: x(active, since 2s) 2024-03-21T21:37:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:17 smithi028 ceph-mon[24806]: from='client.25165 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:37:18.446 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:18 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:18.285+0000 7f8a93d92200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:37:18.446 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:18 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:18.423+0000 7f8a93d92200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:37:19.112 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:18 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:18.798+0000 7f8a93d92200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:37:19.112 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:18 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:18.901+0000 7f8a93d92200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:37:19.112 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:19.111+0000 7f8a93d92200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:37:19.446 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:19.310+0000 7f8a93d92200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:37:19.902 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:19.628+0000 7f8a93d92200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:37:19.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-mon[29568]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:19.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-mon[29568]: mgrmap e35: x(active, since 4s) 2024-03-21T21:37:19.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:19.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:19.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-mon[24806]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:19.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-mon[24806]: mgrmap e35: x(active, since 4s) 2024-03-21T21:37:19.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:19.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:19.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:19 smithi066 ceph-mon[25664]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:19.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:19 smithi066 ceph-mon[25664]: mgrmap e35: x(active, since 4s) 2024-03-21T21:37:20.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:19 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:20.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:19 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:20.152 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:19.901+0000 7f8a93d92200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:37:20.152 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:20.151+0000 7f8a93d92200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:37:21.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:21.033+0000 7f8a93d92200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:37:21.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[29568]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:21.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[24806]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:21.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:21 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:21 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:21 smithi066 ceph-mon[25664]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:21.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:21 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:21 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:21.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:21.648+0000 7f8a93d92200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:37:21.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:21.755+0000 7f8a93d92200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:37:22.392 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:21.993+0000 7f8a93d92200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:37:22.392 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:21 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:37:21] ENGINE Bus STARTING 2024-03-21T21:37:22.392 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CherryPy Checker: 2024-03-21T21:37:22.392 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: The Application mounted at '' has an empty config. 2024-03-21T21:37:22.392 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:37:22.392 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:37:22] ENGINE Serving on http://:::9283 2024-03-21T21:37:22.392 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:37:22] ENGINE Bus STARTED 2024-03-21T21:37:22.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[24806]: Standby manager daemon y started 2024-03-21T21:37:22.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T21:37:22.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:37:22.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T21:37:22.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:37:22.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:22.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:22.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:22.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:37:22.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[29568]: Standby manager daemon y started 2024-03-21T21:37:22.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T21:37:22.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:37:22.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T21:37:22.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:37:22.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:22.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:22.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:22.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:22 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:37:22.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:22 smithi066 ceph-mon[25664]: Standby manager daemon y started 2024-03-21T21:37:22.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:22 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T21:37:22.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:22 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:37:22.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:22 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T21:37:22.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:22 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:37:22.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:22 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:22.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:22 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:22.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:22 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:22.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:22 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:37:23.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:37:23.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:37:23.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: mgrmap e36: x(active, since 8s), standbys: y 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:23.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: mgrmap e36: x(active, since 8s), standbys: y 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:23 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:37:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:37:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: mgrmap e36: x(active, since 8s), standbys: y 2024-03-21T21:37:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:37:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:37:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:37:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:37:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:23.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:23 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:24.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:37:24.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[29568]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T21:37:24.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:37:24.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:37:24.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[29568]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:37:24.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[24806]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:37:24.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[24806]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T21:37:24.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:37:24.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:37:24.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:24 smithi028 ceph-mon[24806]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:37:24.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:24 smithi066 ceph-mon[25664]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:37:24.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:24 smithi066 ceph-mon[25664]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T21:37:24.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:24 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:37:24.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:24 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:37:24.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:24 smithi066 ceph-mon[25664]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:37:26.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:26 smithi028 ceph-mon[24806]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:37:26.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:26 smithi028 ceph-mon[29568]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:37:26.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-mon[25664]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:37:26.491+0000 7fe1ac8f6700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Creating ceph-iscsi config... 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Traceback (most recent call last): 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:37:26.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: return _run_code(code, main_globals, None, 2024-03-21T21:37:26.751 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:37:26.751 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: exec(code, run_globals) 2024-03-21T21:37:26.751 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:37:26.751 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:37:26.751 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:37:26.751 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:37:26.751 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:37:26.751 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:37:26.751 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Traceback (most recent call last): 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: return future.result(timeout) 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:37:26.752 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: return self.__get_result() 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: raise self._exception 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: config_blobs=daemon_spec.final_config, 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Creating ceph-iscsi config... 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: Traceback (most recent call last): 2024-03-21T21:37:26.753 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: return _run_code(code, main_globals, None, 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: exec(code, run_globals) 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:26.754 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.632 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:27 smithi066 ceph-mon[25664]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:37:27.632 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:37:27.632 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:37:27.632 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:37:27.632 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:37:27.632 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.633 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: return future.result(timeout) 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.__get_result() 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: raise self._exception 2024-03-21T21:37:27.634 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:37:27.635 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:27 smithi066 ceph-mon[25664]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:37:27.636 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:27 smithi066 ceph-mon[25664]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:37:27.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:27 smithi028 ceph-mon[24806]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:37:27.696 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:37:27.696 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:37:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: return future.result(timeout) 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:37:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.__get_result() 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: raise self._exception 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:37:27.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:27 smithi028 ceph-mon[24806]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:37:27.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:27 smithi028 ceph-mon[24806]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:27 smithi028 ceph-mon[29568]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:37:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:37:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: return future.result(timeout) 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.__get_result() 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: raise self._exception 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:37:27.703 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:37:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:37:27.705 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:37:27.705 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:37:27.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:27 smithi028 ceph-mon[29568]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:37:27.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:27 smithi028 ceph-mon[29568]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:37:28.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:28 smithi028 ceph-mon[24806]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:37:28.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:28 smithi028 ceph-mon[29568]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:37:28.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:28 smithi066 ceph-mon[25664]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T21:37:30.213 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 systemd[1]: Stopping Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:37:30.213 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 bash[57992]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:37:30.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-mon[25664]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:37:30.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:30.500 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.248Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:37:30.500 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.248Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T21:37:30.500 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.248Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T21:37:30.500 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.249Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T21:37:30.500 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.249Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T21:37:30.500 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.249Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T21:37:30.500 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.249Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T21:37:30.501 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.249Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T21:37:30.501 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.249Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T21:37:30.501 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.249Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T21:37:30.501 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.249Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T21:37:30.501 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[55152]: ts=2024-03-21T21:37:30.249Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T21:37:30.501 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 podman[58005]: 2024-03-21 21:37:30.258901969 +0000 UTC m=+0.125504105 container died 77364ca6b9a3195d75400b11b226d78c6ec411ce7c7bee0ecb0a73cf360a0e48 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:37:30.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:30 smithi028 ceph-mon[24806]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:37:30.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:30 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:30.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:30 smithi028 ceph-mon[29568]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:37:30.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:30 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:30.907 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 podman[58005]: 2024-03-21 21:37:30.623319029 +0000 UTC m=+0.489921164 container cleanup 77364ca6b9a3195d75400b11b226d78c6ec411ce7c7bee0ecb0a73cf360a0e48 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:37:30.907 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 bash[58005]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T21:37:30.907 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 bash[58035]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:37:31.158 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 podman[58020]: 2024-03-21 21:37:30.907950583 +0000 UTC m=+0.651032529 container remove 77364ca6b9a3195d75400b11b226d78c6ec411ce7c7bee0ecb0a73cf360a0e48 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:37:31.158 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T21:37:31.158 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 systemd[1]: Stopped Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:37:31.158 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.439s CPU time. 2024-03-21T21:37:31.158 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:30 smithi066 systemd[1]: Starting Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:37:31.159 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 podman[58101]: 2024-03-21 21:37:31.042931772 +0000 UTC m=+0.020119791 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T21:37:31.159 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 podman[58101]: 2024-03-21 21:37:31.1590046 +0000 UTC m=+0.136192617 container create 8d8f00c2935d8ba4a0b4c2ff2ba4ccbeae37e0b8f7c29935615330b5420cfacb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:37:31.413 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 podman[58101]: 2024-03-21 21:37:31.386115392 +0000 UTC m=+0.363303418 container init 8d8f00c2935d8ba4a0b4c2ff2ba4ccbeae37e0b8f7c29935615330b5420cfacb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:37:31.413 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 podman[58101]: 2024-03-21 21:37:31.388794743 +0000 UTC m=+0.365982762 container start 8d8f00c2935d8ba4a0b4c2ff2ba4ccbeae37e0b8f7c29935615330b5420cfacb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.413Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.413Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.413Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.413Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi066 (none))" 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.413Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.413Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.415Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.416Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.416Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.418Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T21:37:31.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.418Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.375µs 2024-03-21T21:37:31.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.418Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T21:37:31.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 bash[58101]: 8d8f00c2935d8ba4a0b4c2ff2ba4ccbeae37e0b8f7c29935615330b5420cfacb 2024-03-21T21:37:31.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 systemd[1]: Started Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:37:31.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.629Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-03-21T21:37:32.158 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:31.862Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-03-21T21:37:32.159 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:32.100Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-03-21T21:37:32.159 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:32.159Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-03-21T21:37:32.159 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:32.159Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-03-21T21:37:32.159 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:32.159Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=37.454µs wal_replay_duration=741.059436ms total_replay_duration=741.109127ms 2024-03-21T21:37:32.476 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:32.163Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T21:37:32.476 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:32.163Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T21:37:32.477 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:32.163Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T21:37:32.477 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:32.187Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.63422ms db_storage=706ns remote_storage=1.305µs web_handler=410ns query_engine=972ns scrape=483.624µs scrape_sd=131.361µs notify=20.793µs notify_sd=17.919µs rules=22.663732ms 2024-03-21T21:37:32.477 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:37:32.187Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T21:37:32.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-mon[25664]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:37:32.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:32.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:32.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:37:32.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:37:32.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:32 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:37:32.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[29568]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:37:32.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:32.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:32.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:37:32.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:37:32.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:37:32.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[24806]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:37:32.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:32.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:32.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:37:32.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:37:32.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:32 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:37:34.749 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:34 smithi066 ceph-mon[25664]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 483 B/s rd, 0 op/s 2024-03-21T21:37:34.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:34 smithi028 ceph-mon[24806]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 483 B/s rd, 0 op/s 2024-03-21T21:37:34.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:34 smithi028 ceph-mon[29568]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 483 B/s rd, 0 op/s 2024-03-21T21:37:36.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 500 B/s rd, 0 op/s 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:36.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 500 B/s rd, 0 op/s 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:36.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:36 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 500 B/s rd, 0 op/s 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:37:36.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:36.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:36 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:37:38.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:38 smithi028 ceph-mon[29568]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:38.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:38 smithi028 ceph-mon[24806]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:38.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:38 smithi066 ceph-mon[25664]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:40.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:40 smithi028 ceph-mon[24806]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:40.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:40 smithi028 ceph-mon[29568]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:40.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:40 smithi066 ceph-mon[25664]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:41.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:41.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:41.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:37:41.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:37:41.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:37:41.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:37:41.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/4171947009' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/598037530' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2040241574"}]: dispatch 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[24806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/4171947009' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:37:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:41 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/598037530' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2040241574"}]: dispatch 2024-03-21T21:37:41.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:41 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:41 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:41 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:37:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:41 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:37:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:41 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:37:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:41 smithi066 ceph-mon[25664]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:37:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:41 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:37:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:41 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/4171947009' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:37:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:41 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/598037530' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2040241574"}]: dispatch 2024-03-21T21:37:42.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:42 smithi028 ceph-mon[29568]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:42.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:42 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/598037530' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2040241574"}]': finished 2024-03-21T21:37:42.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:42 smithi028 ceph-mon[29568]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T21:37:42.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:42 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/450130908' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4203311745"}]: dispatch 2024-03-21T21:37:42.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:42 smithi028 ceph-mon[24806]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:42.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:42 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/598037530' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2040241574"}]': finished 2024-03-21T21:37:42.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:42 smithi028 ceph-mon[24806]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T21:37:42.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:42 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/450130908' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4203311745"}]: dispatch 2024-03-21T21:37:43.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:37:42] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-03-21T21:37:43.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:42 smithi066 ceph-mon[25664]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:43.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:42 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/598037530' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2040241574"}]': finished 2024-03-21T21:37:43.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:42 smithi066 ceph-mon[25664]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T21:37:43.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:42 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/450130908' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4203311745"}]: dispatch 2024-03-21T21:37:44.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:43 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/450130908' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4203311745"}]': finished 2024-03-21T21:37:44.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:43 smithi066 ceph-mon[25664]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T21:37:44.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:43 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/4134416520' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4203311745"}]: dispatch 2024-03-21T21:37:44.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:43 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/450130908' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4203311745"}]': finished 2024-03-21T21:37:44.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:43 smithi028 ceph-mon[29568]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T21:37:44.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:43 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/4134416520' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4203311745"}]: dispatch 2024-03-21T21:37:44.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:43 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/450130908' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/4203311745"}]': finished 2024-03-21T21:37:44.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:43 smithi028 ceph-mon[24806]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T21:37:44.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:43 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/4134416520' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4203311745"}]: dispatch 2024-03-21T21:37:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[24806]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/4134416520' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4203311745"}]': finished 2024-03-21T21:37:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[24806]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T21:37:45.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2955543881' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2242804737"}]: dispatch 2024-03-21T21:37:45.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2242804737"}]: dispatch 2024-03-21T21:37:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[29568]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/4134416520' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4203311745"}]': finished 2024-03-21T21:37:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[29568]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T21:37:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2955543881' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2242804737"}]: dispatch 2024-03-21T21:37:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:44 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2242804737"}]: dispatch 2024-03-21T21:37:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:44 smithi066 ceph-mon[25664]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:44 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/4134416520' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/4203311745"}]': finished 2024-03-21T21:37:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:44 smithi066 ceph-mon[25664]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T21:37:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:44 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2955543881' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2242804737"}]: dispatch 2024-03-21T21:37:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:44 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2242804737"}]: dispatch 2024-03-21T21:37:46.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:46.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2242804737"}]': finished 2024-03-21T21:37:46.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T21:37:46.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/508073998' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1957706292"}]: dispatch 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1957706292"}]: dispatch 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2242804737"}]': finished 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/508073998' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1957706292"}]: dispatch 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1957706292"}]: dispatch 2024-03-21T21:37:46.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:46.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:46.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:46.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:37:46.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:45 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/2242804737"}]': finished 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/508073998' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1957706292"}]: dispatch 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1957706292"}]: dispatch 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:37:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:45 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:37:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[29568]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[29568]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1957706292"}]': finished 2024-03-21T21:37:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[29568]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T21:37:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1067272493' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3651311135"}]: dispatch 2024-03-21T21:37:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3651311135"}]: dispatch 2024-03-21T21:37:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[24806]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[24806]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1957706292"}]': finished 2024-03-21T21:37:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[24806]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T21:37:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1067272493' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3651311135"}]: dispatch 2024-03-21T21:37:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:46 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3651311135"}]: dispatch 2024-03-21T21:37:47.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:46 smithi066 ceph-mon[25664]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:46 smithi066 ceph-mon[25664]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:37:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:46 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1957706292"}]': finished 2024-03-21T21:37:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:46 smithi066 ceph-mon[25664]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T21:37:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:46 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1067272493' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3651311135"}]: dispatch 2024-03-21T21:37:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:46 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3651311135"}]: dispatch 2024-03-21T21:37:48.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:47 smithi028 ceph-mon[24806]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3651311135"}]': finished 2024-03-21T21:37:48.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:47 smithi028 ceph-mon[24806]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T21:37:48.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:47 smithi028 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3651311135"}]': finished 2024-03-21T21:37:48.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:47 smithi028 ceph-mon[29568]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T21:37:48.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:47 smithi066 ceph-mon[25664]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3651311135"}]': finished 2024-03-21T21:37:48.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:47 smithi066 ceph-mon[25664]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T21:37:49.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:48 smithi028 ceph-mon[24806]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T21:37:49.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:48 smithi028 ceph-mon[29568]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T21:37:49.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:48 smithi066 ceph-mon[25664]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T21:37:51.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:50 smithi028 ceph-mon[24806]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:37:51.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:50 smithi028 ceph-mon[29568]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:37:51.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:50 smithi066 ceph-mon[25664]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:37:52.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:51 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:37:52.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:51 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:37:52.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:51 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:37:52.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:52 smithi066 ceph-mon[25664]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T21:37:53.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:37:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:37:52] "GET /metrics HTTP/1.1" 200 37972 "" "Prometheus/2.33.4" 2024-03-21T21:37:53.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:52 smithi028 ceph-mon[24806]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T21:37:53.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:52 smithi028 ceph-mon[29568]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T21:37:55.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:54 smithi028 ceph-mon[24806]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:55.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:54 smithi028 ceph-mon[29568]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:55.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:54 smithi066 ceph-mon[25664]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:37:57.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:56 smithi028 ceph-mon[24806]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 536 B/s rd, 0 op/s 2024-03-21T21:37:57.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:56 smithi028 ceph-mon[29568]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 536 B/s rd, 0 op/s 2024-03-21T21:37:57.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:56 smithi066 ceph-mon[25664]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 536 B/s rd, 0 op/s 2024-03-21T21:37:59.052 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:58 smithi066 ceph-mon[25664]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-21T21:37:59.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:58 smithi028 ceph-mon[24806]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-21T21:37:59.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:58 smithi028 ceph-mon[29568]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-21T21:38:00.070 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:37:59 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:00.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:37:59 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:00.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:37:59 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:01.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:00 smithi028 ceph-mon[24806]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:01.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:00 smithi028 ceph-mon[29568]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:01.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:00 smithi066 ceph-mon[25664]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:02.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:01 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:02.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:01 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:02.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:01 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:03.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:02 smithi066 ceph-mon[25664]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:03.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:38:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:38:02] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-03-21T21:38:03.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:02 smithi028 ceph-mon[24806]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:03.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:02 smithi028 ceph-mon[29568]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:05.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:04 smithi028 ceph-mon[24806]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:05.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:04 smithi028 ceph-mon[29568]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:05.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:04 smithi066 ceph-mon[25664]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:07.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:06 smithi028 ceph-mon[24806]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:07.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:06 smithi028 ceph-mon[29568]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:07.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:06 smithi066 ceph-mon[25664]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:09.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:08 smithi028 ceph-mon[24806]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:09.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:08 smithi028 ceph-mon[29568]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:09.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:08 smithi066 ceph-mon[25664]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:10.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:10 smithi066 ceph-mon[25664]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:11.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:10 smithi028 ceph-mon[24806]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:11.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:10 smithi028 ceph-mon[29568]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:12.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:11 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:12.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:11 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:12.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:11 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:12.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:12 smithi066 ceph-mon[25664]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:13.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:38:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:38:12] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-03-21T21:38:13.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:12 smithi028 ceph-mon[29568]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:13.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:12 smithi028 ceph-mon[24806]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:15.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:14 smithi028 ceph-mon[24806]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:15.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:14 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:15.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:14 smithi028 ceph-mon[29568]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:15.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:14 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:15.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:14 smithi066 ceph-mon[25664]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:15.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:14 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:17.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:16 smithi028 ceph-mon[24806]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:17.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:16 smithi028 ceph-mon[29568]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:17.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:16 smithi066 ceph-mon[25664]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:19.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:18 smithi028 ceph-mon[24806]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:19.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:18 smithi028 ceph-mon[29568]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:19.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:18 smithi066 ceph-mon[25664]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:21.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:20 smithi028 ceph-mon[24806]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:21.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:20 smithi028 ceph-mon[29568]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:21.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:20 smithi066 ceph-mon[25664]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:22.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:21 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:22.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:21 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:22.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:21 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:23.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:22 smithi066 ceph-mon[25664]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:23.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:38:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:38:22] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-03-21T21:38:23.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:22 smithi028 ceph-mon[24806]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:23.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:22 smithi028 ceph-mon[29568]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:25.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:24 smithi028 ceph-mon[24806]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:25.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:24 smithi028 ceph-mon[29568]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:25.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:24 smithi066 ceph-mon[25664]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:27.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:26 smithi028 ceph-mon[24806]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:27.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:26 smithi028 ceph-mon[29568]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:27.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:26 smithi066 ceph-mon[25664]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:29.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:28 smithi028 ceph-mon[24806]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:28 smithi028 ceph-mon[29568]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:29.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:28 smithi066 ceph-mon[25664]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:30.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:29 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:30.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:29 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:30.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:29 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:30 smithi028 ceph-mon[24806]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:31.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:30 smithi028 ceph-mon[29568]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:31.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:30 smithi066 ceph-mon[25664]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:32.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:31 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:32.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:31 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:32.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:31 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:33.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:32 smithi066 ceph-mon[25664]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:33.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:38:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:38:32] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-03-21T21:38:33.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:32 smithi028 ceph-mon[24806]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:33.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:32 smithi028 ceph-mon[29568]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:35.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:34 smithi028 ceph-mon[24806]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:35.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:34 smithi028 ceph-mon[29568]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:35.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:34 smithi066 ceph-mon[25664]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:37.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:36 smithi028 ceph-mon[24806]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:36 smithi028 ceph-mon[29568]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:37.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:36 smithi066 ceph-mon[25664]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:39.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:38 smithi028 ceph-mon[24806]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:39.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:38 smithi028 ceph-mon[29568]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:39.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:38 smithi066 ceph-mon[25664]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:41.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:40 smithi028 ceph-mon[24806]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:41.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:40 smithi028 ceph-mon[29568]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:41.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:40 smithi066 ceph-mon[25664]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:42.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:41 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:42.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:41 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:42.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:41 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:42.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:42 smithi066 ceph-mon[25664]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:43.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:38:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:38:42] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-03-21T21:38:43.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:42 smithi028 ceph-mon[24806]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:43.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:42 smithi028 ceph-mon[29568]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:38:45.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:44 smithi028 ceph-mon[29568]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:45.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:44 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:44 smithi028 ceph-mon[24806]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:44 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:45.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:44 smithi066 ceph-mon[25664]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:45.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:44 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:38:46.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:45 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:38:46.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:45 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:38:46.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:45 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:38:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:46 smithi028 ceph-mon[29568]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:46 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:38:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:46 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:38:47.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:46 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:38:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:46 smithi028 ceph-mon[24806]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:46 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:38:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:46 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:38:47.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:46 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:38:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:46 smithi066 ceph-mon[25664]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:38:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:46 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:38:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:46 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:38:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:46 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:38:48.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:47 smithi028 ceph-mon[24806]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T21:38:48.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:47 smithi028 ceph-mon[24806]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:48.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:47 smithi028 ceph-mon[29568]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T21:38:48.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:47 smithi028 ceph-mon[29568]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:48.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:47 smithi066 ceph-mon[25664]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T21:38:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:47 smithi066 ceph-mon[25664]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:50.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:49 smithi028 ceph-mon[29568]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:50.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:49 smithi028 ceph-mon[24806]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:50.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:49 smithi066 ceph-mon[25664]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:52.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:51 smithi028 ceph-mon[29568]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-03-21T21:38:52.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:51 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:52.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:51 smithi028 ceph-mon[24806]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-03-21T21:38:52.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:51 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:52.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:51 smithi066 ceph-mon[25664]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-03-21T21:38:52.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:51 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:38:53.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:38:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:38:52] "GET /metrics HTTP/1.1" 200 37972 "" "Prometheus/2.33.4" 2024-03-21T21:38:54.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:53 smithi028 ceph-mon[24806]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:54.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:53 smithi028 ceph-mon[29568]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:54.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:53 smithi066 ceph-mon[25664]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:56.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:55 smithi028 ceph-mon[24806]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:56.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:55 smithi028 ceph-mon[29568]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:56.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:55 smithi066 ceph-mon[25664]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:38:58.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:57 smithi028 ceph-mon[29568]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:38:58.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:57 smithi028 ceph-mon[24806]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:38:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:57 smithi066 ceph-mon[25664]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:39:00.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:59 smithi028 ceph-mon[29568]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:00.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:38:59 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:00.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:59 smithi028 ceph-mon[24806]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:00.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:38:59 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:00.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:59 smithi066 ceph-mon[25664]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:00.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:38:59 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:02.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:01 smithi028 ceph-mon[29568]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:02.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:01 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:02.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:01 smithi028 ceph-mon[24806]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:02.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:01 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:02.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:01 smithi066 ceph-mon[25664]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:02.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:01 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:03.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:39:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:39:02] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-21T21:39:04.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:03 smithi028 ceph-mon[24806]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:04.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:03 smithi028 ceph-mon[29568]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:04.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:03 smithi066 ceph-mon[25664]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:06.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:05 smithi028 ceph-mon[24806]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:06.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:05 smithi028 ceph-mon[29568]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:06.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:05 smithi066 ceph-mon[25664]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:08.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:07 smithi028 ceph-mon[24806]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:08.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:07 smithi028 ceph-mon[29568]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:08.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:07 smithi066 ceph-mon[25664]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:10.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:09 smithi028 ceph-mon[24806]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:10.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:09 smithi028 ceph-mon[29568]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:10.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:09 smithi066 ceph-mon[25664]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:12.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:11 smithi028 ceph-mon[29568]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:12.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:11 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:12.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:11 smithi028 ceph-mon[24806]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:11 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:12.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:11 smithi066 ceph-mon[25664]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:12.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:11 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:13.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:39:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:39:12] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-21T21:39:14.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:13 smithi028 ceph-mon[29568]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:14.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:13 smithi028 ceph-mon[24806]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:14.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:13 smithi066 ceph-mon[25664]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:15.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:14 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:15.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:14 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:15.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:14 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:16.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:15 smithi028 ceph-mon[29568]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:15 smithi028 ceph-mon[24806]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:15 smithi066 ceph-mon[25664]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:18.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:17 smithi028 ceph-mon[29568]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:18.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:17 smithi028 ceph-mon[24806]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:18.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:17 smithi066 ceph-mon[25664]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:20.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:19 smithi028 ceph-mon[29568]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:20.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:19 smithi028 ceph-mon[24806]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:20.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:19 smithi066 ceph-mon[25664]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:22.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:21 smithi028 ceph-mon[29568]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:22.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:21 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:21 smithi028 ceph-mon[24806]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:21 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:22.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:21 smithi066 ceph-mon[25664]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:22.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:21 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:23.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:39:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:39:22] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-21T21:39:24.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:23 smithi028 ceph-mon[29568]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:24.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:23 smithi028 ceph-mon[24806]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:24.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:23 smithi066 ceph-mon[25664]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:26.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:25 smithi028 ceph-mon[29568]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:25 smithi028 ceph-mon[24806]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:26.249 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:25 smithi066 ceph-mon[25664]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:28.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:27 smithi066 ceph-mon[25664]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:28.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:27 smithi028 ceph-mon[29568]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:28.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:27 smithi028 ceph-mon[24806]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:30.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:29 smithi066 ceph-mon[25664]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:30.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:29 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:30.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:29 smithi028 ceph-mon[29568]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:30.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:29 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:30.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:29 smithi028 ceph-mon[24806]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:30.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:29 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:32.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:31 smithi066 ceph-mon[25664]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:32.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:31 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:32.347 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:31 smithi028 ceph-mon[29568]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:32.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:31 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:32.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:31 smithi028 ceph-mon[24806]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:32.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:31 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:33.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:39:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:39:32] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T21:39:34.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:33 smithi066 ceph-mon[25664]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:33 smithi028 ceph-mon[24806]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:33 smithi028 ceph-mon[29568]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:36.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:35 smithi066 ceph-mon[25664]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:36.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:35 smithi028 ceph-mon[24806]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:36.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:35 smithi028 ceph-mon[29568]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:38.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:37 smithi066 ceph-mon[25664]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:37 smithi028 ceph-mon[24806]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:38.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:37 smithi028 ceph-mon[29568]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:40.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:39 smithi066 ceph-mon[25664]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:40.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:39 smithi028 ceph-mon[24806]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:40.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:39 smithi028 ceph-mon[29568]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:42.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:41 smithi066 ceph-mon[25664]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:42.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:41 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:42.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:41 smithi028 ceph-mon[29568]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:42.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:41 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:41 smithi028 ceph-mon[24806]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:41 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:43.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:39:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:39:42] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T21:39:44.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:43 smithi066 ceph-mon[25664]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:44.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:43 smithi028 ceph-mon[29568]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:44.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:43 smithi028 ceph-mon[24806]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:44 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:45.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:44 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:45.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:44 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:39:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:45 smithi066 ceph-mon[25664]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:46.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:45 smithi028 ceph-mon[29568]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:46.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:45 smithi028 ceph-mon[24806]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:46 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:39:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:46 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:39:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:46 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:39:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:46 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:39:47.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:46 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:39:47.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:46 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:39:47.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:46 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:39:47.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:46 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:39:47.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:46 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:39:47.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:46 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:39:47.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:46 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:39:47.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:46 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:39:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:47 smithi066 ceph-mon[25664]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:47 smithi066 ceph-mon[25664]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-21T21:39:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:47 smithi066 ceph-mon[25664]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:48.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:47 smithi028 ceph-mon[24806]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:48.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:47 smithi028 ceph-mon[24806]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-21T21:39:48.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:47 smithi028 ceph-mon[24806]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:48.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:47 smithi028 ceph-mon[29568]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:39:48.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:47 smithi028 ceph-mon[29568]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-21T21:39:48.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:47 smithi028 ceph-mon[29568]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:49.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:49 smithi066 ceph-mon[25664]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:49.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:49 smithi028 ceph-mon[24806]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:49.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:49 smithi028 ceph-mon[29568]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:52.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:51 smithi066 ceph-mon[25664]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:52.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:51 smithi028 ceph-mon[24806]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:52.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:51 smithi028 ceph-mon[29568]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:53.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:52 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:53.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:39:52 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:39:52] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-21T21:39:53.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:52 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:53.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:52 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:39:54.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:53 smithi066 ceph-mon[25664]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:54.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:53 smithi028 ceph-mon[29568]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:54.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:53 smithi028 ceph-mon[24806]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:39:56.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:55 smithi066 ceph-mon[25664]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-21T21:39:56.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:55 smithi028 ceph-mon[29568]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-21T21:39:56.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:55 smithi028 ceph-mon[24806]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-21T21:39:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:57 smithi066 ceph-mon[25664]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:39:58.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:57 smithi028 ceph-mon[29568]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:39:58.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:57 smithi028 ceph-mon[24806]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:39:59.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:39:59 smithi066 ceph-mon[25664]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:59.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:39:59 smithi028 ceph-mon[24806]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:39:59.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:39:59 smithi028 ceph-mon[29568]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:00.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:00 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:00.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:00 smithi066 ceph-mon[25664]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T21:40:00.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:00 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:00.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:00 smithi028 ceph-mon[24806]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T21:40:00.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:00 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:00.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:00 smithi028 ceph-mon[29568]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T21:40:01.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:01 smithi066 ceph-mon[25664]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:01.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:01 smithi028 ceph-mon[24806]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:01.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:01 smithi028 ceph-mon[29568]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:02.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:02 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:02.848 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:02 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:02.848 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:02 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:03.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:40:02] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-03-21T21:40:03.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:03 smithi066 ceph-mon[25664]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:03.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:03 smithi028 ceph-mon[24806]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:03.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:03 smithi028 ceph-mon[29568]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:06.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:05 smithi066 ceph-mon[25664]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:06.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:05 smithi028 ceph-mon[29568]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:06.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:05 smithi028 ceph-mon[24806]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:08.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:07 smithi066 ceph-mon[25664]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:08.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:07 smithi028 ceph-mon[29568]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:08.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:07 smithi028 ceph-mon[24806]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:09.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:09 smithi066 ceph-mon[25664]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:09.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:09 smithi028 ceph-mon[24806]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:09 smithi028 ceph-mon[29568]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:12.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:11 smithi066 ceph-mon[25664]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:12.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:11 smithi028 ceph-mon[24806]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:12.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:11 smithi028 ceph-mon[29568]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:13.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:12 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:13.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:40:12] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-03-21T21:40:13.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:12 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:13.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:12 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:14.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:13 smithi066 ceph-mon[25664]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:14.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:13 smithi028 ceph-mon[24806]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:14.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:13 smithi028 ceph-mon[29568]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:15.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:14 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:14 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:15.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:14 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:15 smithi066 ceph-mon[25664]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:16.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:15 smithi028 ceph-mon[24806]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:16.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:15 smithi028 ceph-mon[29568]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:17.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:40:18.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:17 smithi028 ceph-mon[29568]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:18.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:17 smithi028 ceph-mon[24806]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:18.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:17 smithi066 ceph-mon[25664]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:19.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:19 smithi028 ceph-mon[29568]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:19.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:19 smithi028 ceph-mon[24806]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:19.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:19 smithi066 ceph-mon[25664]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:19.977 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:40:19.977 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (12m) 2m ago 20m 22.7M - 0.23.0 ba2b418f427c cd065b97fad1 2024-03-21T21:40:19.977 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 error 2m ago 20m - - 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (2m) 2m ago 19m 57.0M - 3.5 e1d6a67b021e 48135ddcc26c 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283 running (15m) 2m ago 25m 496M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 7fe31a1b09b6 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (9m) 2m ago 26m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (26m) 2m ago 27m 72.4M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (25m) 2m ago 25m 54.0M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (25m) 2m ago 25m 52.2M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (12m) 2m ago 21m 21.5M - 1.3.1 1dbe0e931976 a8a424c65ebe 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (12m) 2m ago 21m 19.0M - 1.3.1 1dbe0e931976 f6cdb9cb9923 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (24m) 2m ago 24m 52.9M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (24m) 2m ago 24m 53.6M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (23m) 2m ago 23m 52.3M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (23m) 2m ago 23m 51.8M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:40:19.978 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (23m) 2m ago 23m 52.4M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:40:19.979 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (22m) 2m ago 22m 51.6M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:40:19.979 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (22m) 2m ago 22m 51.3M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:40:19.979 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (22m) 2m ago 22m 52.2M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:40:19.979 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (2m) 2m ago 21m 42.3M - 2.33.4 514e6a882f6e 8d8f00c2935d 2024-03-21T21:40:19.979 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (19m) 2m ago 19m 75.9M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:40:19.979 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (19m) 2m ago 19m 74.2M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:40:20.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:20 smithi028 ceph-mon[29568]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:20.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:20 smithi028 ceph-mon[24806]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:20.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ls' 2024-03-21T21:40:20.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:20 smithi066 ceph-mon[25664]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:21.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:21 smithi066 ceph-mon[25664]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:21.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:21 smithi028 ceph-mon[24806]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:21.913 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:21 smithi028 ceph-mon[29568]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:22.612 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-21T21:40:22.612 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager ?:9093,9094 1/1 2m ago 21m smithi028=a;count:1 2024-03-21T21:40:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:grafana ?:3000 0/1 2m ago 21m smithi066=a;count:1 2024-03-21T21:40:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo ?:5000 1/1 2m ago 19m count:1 2024-03-21T21:40:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 2/2 2m ago 25m smithi028=y;smithi066=x;count:2 2024-03-21T21:40:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 2m ago 25m smithi028:172.21.15.28=a;smithi028:[v2:172.21.15.28:3301,v1:172.21.15.28:6790]=c;smithi066:172.21.15.66=b;count:3 2024-03-21T21:40:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter ?:9100 2/2 2m ago 21m smithi028=a;smithi066=b;count:2 2024-03-21T21:40:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:osd 8 2m ago - 2024-03-21T21:40:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus ?:9095 1/1 2m ago 21m smithi066=a;count:1 2024-03-21T21:40:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo ?:8000 2/2 2m ago 19m count:2 2024-03-21T21:40:22.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:22 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:22.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:22 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:22 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:23.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:40:22] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-21T21:40:23.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions' 2024-03-21T21:40:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:23 smithi066 ceph-mon[25664]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:23.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:23 smithi066 ceph-mon[25664]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:23.751 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:23 smithi028 ceph-mon[29568]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:23.751 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:23 smithi028 ceph-mon[29568]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:23.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:23 smithi028 ceph-mon[24806]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:23.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:23 smithi028 ceph-mon[24806]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:25.391 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:40:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:40:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:40:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:40:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:40:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:40:25.392 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:40:26.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-21T21:40:26.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:25 smithi066 ceph-mon[25664]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:26.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:25 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/4121568301' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:40:26.309 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:25 smithi028 ceph-mon[29568]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:26.309 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:25 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/4121568301' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:25 smithi028 ceph-mon[24806]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:25 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/4121568301' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:40:28.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:27 smithi066 ceph-mon[25664]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:28.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:27 smithi028 ceph-mon[29568]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:28.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:27 smithi028 ceph-mon[24806]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:40:29.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:29 smithi028 ceph-mon[29568]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:29.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:29 smithi028 ceph-mon[29568]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:29 smithi028 ceph-mon[24806]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:29 smithi028 ceph-mon[24806]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:29.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:29 smithi066 ceph-mon[25664]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:29.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:29 smithi066 ceph-mon[25664]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:40:30.127 INFO:teuthology.orchestra.run.smithi028.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:40:30.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:30.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:40:30.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:30.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:30.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:40:30.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:40:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:40:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:40:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:30 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:40:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:30 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:30 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:40:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:30 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:30 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:30 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:40:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:30 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:40:30.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T21:40:31.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:31 smithi028 ceph-mon[29568]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:40:31.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:31 smithi028 ceph-mon[29568]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-21T21:40:31.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:31 smithi028 ceph-mon[29568]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-03-21T21:40:31.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:31 smithi028 ceph-mon[29568]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:40:31.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:31 smithi028 ceph-mon[24806]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:40:31.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:31 smithi028 ceph-mon[24806]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-21T21:40:31.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:31 smithi028 ceph-mon[24806]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-03-21T21:40:31.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:31 smithi028 ceph-mon[24806]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:40:31.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:31 smithi066 ceph-mon[25664]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:40:31.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:31 smithi066 ceph-mon[25664]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-21T21:40:31.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:31 smithi066 ceph-mon[25664]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-03-21T21:40:31.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:31 smithi066 ceph-mon[25664]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:40:32.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:32 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:32.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:32 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:32.748 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:32 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:33.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:40:32] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-21T21:40:33.563 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:33 smithi028 ceph-mon[29568]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:40:33.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:33 smithi028 ceph-mon[24806]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:40:33.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:33 smithi066 ceph-mon[25664]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:40:34.164 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:40:34.983 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:34.714+0000 7fe21a286700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-21T21:40:34.983 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ignoring --setuser ceph since I am not root 2024-03-21T21:40:34.983 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ignoring --setgroup ceph since I am not root 2024-03-21T21:40:34.983 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mgr[45923]: -- 172.21.15.66:0/3931658787 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564b24a865a0 con 0x564b24a5a800 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: Failing over to other MGR 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T21:40:34.984 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:34 smithi066 ceph-mon[25664]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T21:40:35.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:40:35.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: Failing over to other MGR 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[24806]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' 2024-03-21T21:40:35.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: Failing over to other MGR 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: from='mgr.15231 172.21.15.66:0/145541910' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-mon[29568]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:40:34] ENGINE Bus STOPPING 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:40:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:40:35.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:40:34] ENGINE Bus STOPPED 2024-03-21T21:40:35.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:40:34] ENGINE Bus STARTING 2024-03-21T21:40:35.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:40:34] ENGINE Serving on http://:::9283 2024-03-21T21:40:35.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:40:34] ENGINE Bus STARTED 2024-03-21T21:40:35.719 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:35.322+0000 7f9ecdbad200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:40:35.720 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:35.421+0000 7f9ecdbad200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:40:35.756 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:40:35.756 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (12m) 2m ago 20m 22.7M - 0.23.0 ba2b418f427c cd065b97fad1 2024-03-21T21:40:35.756 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 error 3m ago 20m - - 2024-03-21T21:40:35.756 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (2m) 2m ago 19m 57.0M - 3.5 e1d6a67b021e 48135ddcc26c 2024-03-21T21:40:35.756 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283 running (15m) 3m ago 25m 496M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 7fe31a1b09b6 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (9m) 2m ago 27m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (27m) 2m ago 27m 72.4M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (25m) 3m ago 25m 54.0M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (25m) 2m ago 25m 52.2M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (12m) 2m ago 21m 21.5M - 1.3.1 1dbe0e931976 a8a424c65ebe 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (12m) 3m ago 21m 19.0M - 1.3.1 1dbe0e931976 f6cdb9cb9923 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (25m) 2m ago 25m 52.9M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (24m) 2m ago 24m 53.6M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (24m) 2m ago 24m 52.3M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (23m) 2m ago 23m 51.8M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (23m) 3m ago 23m 52.4M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (23m) 3m ago 23m 51.6M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (22m) 3m ago 22m 51.3M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (22m) 3m ago 22m 52.2M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:40:35.757 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (3m) 3m ago 21m 42.3M - 2.33.4 514e6a882f6e 8d8f00c2935d 2024-03-21T21:40:35.758 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (20m) 2m ago 20m 75.9M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:40:35.758 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (19m) 3m ago 19m 74.2M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:40:35.993 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.15231 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: mgrmap e37: y(active, starting, since 0.995129s) 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:40:35.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: Manager daemon y is now available 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.15231 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: mgrmap e37: y(active, starting, since 0.995129s) 2024-03-21T21:40:35.995 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:40:35.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: Manager daemon y is now available 2024-03-21T21:40:35.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:35.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:35.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:40:35.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.15231 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: mgrmap e37: y(active, starting, since 0.995129s) 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:40:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:40:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:40:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:40:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:40:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:40:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: Manager daemon y is now available 2024-03-21T21:40:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:40:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:40:36.190 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:40:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:40:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:40:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:40:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:40:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 1 2024-03-21T21:40:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:40:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 1 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:40:36.191 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:40:36.264 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:36.174+0000 7f9ecdbad200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:40:36.533 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:36.272+0000 7f9ecdbad200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:40:36.533 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:36.420+0000 7f9ecdbad200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:40:36.533 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:36.531+0000 7f9ecdbad200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:40:36.544 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr" 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:40:36.545 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:40:36.850 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:36 smithi066 ceph-mon[25664]: mgrmap e38: y(active, since 2s) 2024-03-21T21:40:36.850 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:36 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3506672784' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:40:36.851 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:36.620+0000 7f9ecdbad200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:40:36.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:36 smithi028 ceph-mon[29568]: mgrmap e38: y(active, since 2s) 2024-03-21T21:40:36.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:36 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3506672784' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:40:36.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:36 smithi028 ceph-mon[24806]: mgrmap e38: y(active, since 2s) 2024-03-21T21:40:36.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:36 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3506672784' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:40:37.114 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:36.902+0000 7f9ecdbad200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T21:40:37.114 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:37.022+0000 7f9ecdbad200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:40:37.114 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:37.113+0000 7f9ecdbad200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:40:37.409 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:37.212+0000 7f9ecdbad200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:40:37.409 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:37.407+0000 7f9ecdbad200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:40:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-mon[25664]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-mon[25664]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-mon[25664]: from='client.15471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-mon[25664]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-mon[25664]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:40:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:37 smithi066 ceph-mon[25664]: Cluster is now healthy 2024-03-21T21:40:38.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[29568]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:38.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[29568]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:38.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[29568]: from='client.15471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:38.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[29568]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:38.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[29568]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:40:38.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[29568]: Cluster is now healthy 2024-03-21T21:40:38.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[24806]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:38.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[24806]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:38.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[24806]: from='client.15471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:40:38.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[24806]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:38.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[24806]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:40:38.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:37 smithi028 ceph-mon[24806]: Cluster is now healthy 2024-03-21T21:40:38.737 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:38.465+0000 7f9ecdbad200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:40:38.737 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:38.612+0000 7f9ecdbad200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:40:38.841 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[29568]: [21/Mar/2024:21:40:36] ENGINE Bus STARTING 2024-03-21T21:40:38.841 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[29568]: [21/Mar/2024:21:40:36] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:40:38.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[29568]: [21/Mar/2024:21:40:36] ENGINE Serving on http://172.21.15.28:8765 2024-03-21T21:40:38.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[29568]: [21/Mar/2024:21:40:36] ENGINE Bus STARTED 2024-03-21T21:40:38.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[29568]: mgrmap e39: y(active, since 4s) 2024-03-21T21:40:38.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[24806]: [21/Mar/2024:21:40:36] ENGINE Bus STARTING 2024-03-21T21:40:38.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[24806]: [21/Mar/2024:21:40:36] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:40:38.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[24806]: [21/Mar/2024:21:40:36] ENGINE Serving on http://172.21.15.28:8765 2024-03-21T21:40:38.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[24806]: [21/Mar/2024:21:40:36] ENGINE Bus STARTED 2024-03-21T21:40:38.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:38 smithi028 ceph-mon[24806]: mgrmap e39: y(active, since 4s) 2024-03-21T21:40:39.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:38 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:38.978+0000 7f9ecdbad200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:40:39.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:38 smithi066 ceph-mon[25664]: [21/Mar/2024:21:40:36] ENGINE Bus STARTING 2024-03-21T21:40:39.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:38 smithi066 ceph-mon[25664]: [21/Mar/2024:21:40:36] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:40:39.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:38 smithi066 ceph-mon[25664]: [21/Mar/2024:21:40:36] ENGINE Serving on http://172.21.15.28:8765 2024-03-21T21:40:39.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:38 smithi066 ceph-mon[25664]: [21/Mar/2024:21:40:36] ENGINE Bus STARTED 2024-03-21T21:40:39.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:38 smithi066 ceph-mon[25664]: mgrmap e39: y(active, since 4s) 2024-03-21T21:40:39.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:39.097+0000 7f9ecdbad200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:40:39.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:39.317+0000 7f9ecdbad200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:40:39.837 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:39.523+0000 7f9ecdbad200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:40:39.837 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:39 smithi066 ceph-mon[25664]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:39.838 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:39 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:39.838 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:39 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:40.117 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:39.836+0000 7f9ecdbad200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:40:40.117 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:40 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:40.116+0000 7f9ecdbad200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:40:40.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:39 smithi028 ceph-mon[29568]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:40.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:39 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:40.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:39 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:40.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:39 smithi028 ceph-mon[24806]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:40.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:39 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:40.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:39 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:40.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:40 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:40.373+0000 7f9ecdbad200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:40:41.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[29568]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:41.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:40 smithi028 ceph-mon[24806]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:41.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:40 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:40 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:40 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:40 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:41.242 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:40 smithi066 ceph-mon[25664]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:41.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:41.241+0000 7f9ecdbad200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:40:42.203 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:41.852+0000 7f9ecdbad200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:40:42.203 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:41.960+0000 7f9ecdbad200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:40:42.203 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:42.202+0000 7f9ecdbad200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:40:42.203 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.203 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.203 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:42.203 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:40:42.203 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:40:42.203 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.204 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='client.15351 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:40:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:40:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:40:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:42 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:42.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:40:42] ENGINE Bus STARTING 2024-03-21T21:40:42.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: CherryPy Checker: 2024-03-21T21:40:42.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: The Application mounted at '' has an empty config. 2024-03-21T21:40:42.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: 2024-03-21T21:40:42.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:40:42] ENGINE Serving on http://:::9283 2024-03-21T21:40:42.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: [21/Mar/2024:21:40:42] ENGINE Bus STARTED 2024-03-21T21:40:43.000 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[45919]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:40:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T21:40:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:40:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 61 op/s 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: Standby manager daemon x started 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[24806]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:40:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 61 op/s 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: Standby manager daemon x started 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:40:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:43 smithi028 ceph-mon[29568]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:40:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:40:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 61 op/s 2024-03-21T21:40:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:40:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:40:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:40:43.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: Standby manager daemon x started 2024-03-21T21:40:43.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:40:43.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:40:43.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:40:43.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/3150843722' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:40:43.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:43 smithi066 ceph-mon[25664]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:40:44.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:44 smithi028 ceph-mon[29568]: mgrmap e40: y(active, since 9s), standbys: x 2024-03-21T21:40:44.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:44 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:40:44.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:44 smithi028 ceph-mon[24806]: mgrmap e40: y(active, since 9s), standbys: x 2024-03-21T21:40:44.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:44 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:40:44.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:44 smithi066 ceph-mon[25664]: mgrmap e40: y(active, since 9s), standbys: x 2024-03-21T21:40:44.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:44 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:40:45.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-mon[24806]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 46 op/s 2024-03-21T21:40:45.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-mon[29568]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 46 op/s 2024-03-21T21:40:45.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:40:45.430+0000 7f8a19e4f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:40:45.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:40:45.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return future.result(timeout) 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.__get_result() 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: raise self._exception 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: config_blobs=daemon_spec.final_config, 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:40:45.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:40:45.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:40:45.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:45.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:40:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:45.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:45 smithi066 ceph-mon[25664]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 46 op/s 2024-03-21T21:40:46.545 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:46 smithi066 ceph-mon[25664]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:40:46.545 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:40:46.545 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:40:46.546 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: return future.result(timeout) 2024-03-21T21:40:46.547 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.__get_result() 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: raise self._exception 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:40:46.548 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:40:46.549 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.550 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.550 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:46 smithi066 ceph-mon[25664]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:40:46.550 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:46 smithi066 ceph-mon[25664]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:40:46.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:46 smithi028 ceph-mon[24806]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:40:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:40:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: return future.result(timeout) 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.__get_result() 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: raise self._exception 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:40:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:40:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:46 smithi028 ceph-mon[29568]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:40:46.951 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:40:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:40:46.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: return future.result(timeout) 2024-03-21T21:40:46.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:40:46.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.__get_result() 2024-03-21T21:40:46.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: raise self._exception 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:40:46.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:40:46.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:40:46.956 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:40:46.956 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:40:46.956 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:40:46.956 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.956 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:46 smithi028 ceph-mon[24806]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:40:46.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:46 smithi028 ceph-mon[24806]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:40:46.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:40:46.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:40:46.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:40:46.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:40:46.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:40:46.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:46 smithi028 ceph-mon[29568]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T21:40:46.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:46 smithi028 ceph-mon[29568]: Reconfiguring daemon prometheus.a on smithi066 2024-03-21T21:40:47.711 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:47 smithi066 ceph-mon[25664]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 107 op/s 2024-03-21T21:40:47.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:47 smithi028 ceph-mon[24806]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 107 op/s 2024-03-21T21:40:47.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:47 smithi028 ceph-mon[29568]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 107 op/s 2024-03-21T21:40:49.097 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:48 smithi066 systemd[1]: Stopping Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:40:49.098 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:48 smithi066 bash[60843]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:40:49.381 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.096Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:40:49.381 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.097Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T21:40:49.381 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.097Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T21:40:49.381 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.097Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T21:40:49.381 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.097Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T21:40:49.381 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.097Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T21:40:49.381 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.097Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T21:40:49.382 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.097Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T21:40:49.382 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.097Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T21:40:49.382 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.098Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T21:40:49.382 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.098Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T21:40:49.382 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[58116]: ts=2024-03-21T21:40:49.098Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T21:40:49.382 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 podman[60856]: 2024-03-21 21:40:49.107120727 +0000 UTC m=+0.113420215 container died 8d8f00c2935d8ba4a0b4c2ff2ba4ccbeae37e0b8f7c29935615330b5420cfacb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:40:49.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:49 smithi066 ceph-mon[25664]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 98 op/s 2024-03-21T21:40:49.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 podman[60856]: 2024-03-21 21:40:49.496392431 +0000 UTC m=+0.502691928 container cleanup 8d8f00c2935d8ba4a0b4c2ff2ba4ccbeae37e0b8f7c29935615330b5420cfacb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:40:49.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 bash[60856]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T21:40:49.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 bash[60882]: Error: no container with name or ID "ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T21:40:49.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:49 smithi028 ceph-mon[24806]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 98 op/s 2024-03-21T21:40:49.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:49 smithi028 ceph-mon[29568]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 98 op/s 2024-03-21T21:40:50.030 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 podman[60871]: 2024-03-21 21:40:49.778935639 +0000 UTC m=+0.673584134 container remove 8d8f00c2935d8ba4a0b4c2ff2ba4ccbeae37e0b8f7c29935615330b5420cfacb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:40:50.030 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T21:40:50.030 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 systemd[1]: Stopped Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:40:50.030 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.491s CPU time. 2024-03-21T21:40:50.030 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:49 smithi066 systemd[1]: Starting Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:40:50.030 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 podman[60953]: 2024-03-21 21:40:49.925082574 +0000 UTC m=+0.019890482 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T21:40:50.030 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 podman[60953]: 2024-03-21 21:40:50.03003988 +0000 UTC m=+0.124847765 container create fe8071f8b95eed5094f2316aef121a2c7f8717e66f9b0e8a319afc9bcade9195 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 podman[60953]: 2024-03-21 21:40:50.2406038 +0000 UTC m=+0.335411707 container init fe8071f8b95eed5094f2316aef121a2c7f8717e66f9b0e8a319afc9bcade9195 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 podman[60953]: 2024-03-21 21:40:50.243252462 +0000 UTC m=+0.338060347 container start fe8071f8b95eed5094f2316aef121a2c7f8717e66f9b0e8a319afc9bcade9195 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.268Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.268Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.268Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.268Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi066 (none))" 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.268Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.268Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.269Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.270Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T21:40:50.341 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.271Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T21:40:50.342 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.272Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T21:40:50.342 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.272Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.582µs 2024-03-21T21:40:50.342 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.272Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T21:40:50.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:50.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:50.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:50.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:40:50.741 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:50.742 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 bash[60953]: fe8071f8b95eed5094f2316aef121a2c7f8717e66f9b0e8a319afc9bcade9195 2024-03-21T21:40:50.742 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 systemd[1]: Started Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:40:50.742 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.470Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-03-21T21:40:50.742 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.741Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-03-21T21:40:50.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:50.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:50.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:50.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:40:50.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:50.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:40:50.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:50.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:50.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:40:50.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:51.250 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:50 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:50.998Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-03-21T21:40:51.250 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:51.050Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-03-21T21:40:51.251 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:51.058Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-03-21T21:40:51.251 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:51.058Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-03-21T21:40:51.251 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:51.058Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=44.052µs wal_replay_duration=786.209135ms total_replay_duration=786.267945ms 2024-03-21T21:40:51.251 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:51.062Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T21:40:51.251 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:51.062Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T21:40:51.251 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:51.062Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T21:40:51.251 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:51.083Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.825797ms db_storage=811ns remote_storage=1.511µs web_handler=360ns query_engine=735ns scrape=433.871µs scrape_sd=117.094µs notify=19.656µs notify_sd=12.231µs rules=19.822491ms 2024-03-21T21:40:51.251 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:40:51.083Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T21:40:51.602 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-mon[25664]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 98 op/s 2024-03-21T21:40:51.602 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:51 smithi066 ceph-mon[25664]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:40:51.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:51 smithi028 ceph-mon[29568]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 98 op/s 2024-03-21T21:40:51.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:51 smithi028 ceph-mon[29568]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:40:51.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:51 smithi028 ceph-mon[24806]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 98 op/s 2024-03-21T21:40:51.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:51 smithi028 ceph-mon[24806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:40:52.687 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:52 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:40:52.687 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:52 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:40:52.687 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:52 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:52 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:40:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:52 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:40:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:52 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:52.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:52 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:40:52.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:52 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:40:52.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:52 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:40:53.704 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:53 smithi066 ceph-mon[25664]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 98 op/s 2024-03-21T21:40:53.704 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:53 smithi066 ceph-mon[25664]: Upgrade: Updating mgr.x 2024-03-21T21:40:53.704 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:53 smithi066 ceph-mon[25664]: Deploying daemon mgr.x on smithi066 2024-03-21T21:40:53.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:53 smithi028 ceph-mon[24806]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 98 op/s 2024-03-21T21:40:53.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:53 smithi028 ceph-mon[24806]: Upgrade: Updating mgr.x 2024-03-21T21:40:53.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:53 smithi028 ceph-mon[24806]: Deploying daemon mgr.x on smithi066 2024-03-21T21:40:53.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:53 smithi028 ceph-mon[29568]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 98 op/s 2024-03-21T21:40:53.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:53 smithi028 ceph-mon[29568]: Upgrade: Updating mgr.x 2024-03-21T21:40:53.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:53 smithi028 ceph-mon[29568]: Deploying daemon mgr.x on smithi066 2024-03-21T21:40:54.281 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:54 smithi066 systemd[1]: Stopping Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:40:54.550 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:54 smithi066 podman[61307]: 2024-03-21 21:40:54.280745383 +0000 UTC m=+0.136416030 container died 7fe31a1b09b61c385055e587f9e2de0812ae2c00577b58b78cb1be31e3f71f73 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:40:54.825 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:54 smithi066 podman[61307]: 2024-03-21 21:40:54.616169738 +0000 UTC m=+0.471840371 container cleanup 7fe31a1b09b61c385055e587f9e2de0812ae2c00577b58b78cb1be31e3f71f73 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-21T21:40:54.826 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:54 smithi066 bash[61307]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x 2024-03-21T21:40:55.229 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:54 smithi066 podman[61321]: 2024-03-21 21:40:54.82567706 +0000 UTC m=+0.543415130 container remove 7fe31a1b09b61c385055e587f9e2de0812ae2c00577b58b78cb1be31e3f71f73 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-21T21:40:55.229 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:54 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:40:55.229 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-21T21:40:55.230 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 systemd[1]: Stopped Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:40:55.230 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service: Consumed 1min 2.622s CPU time. 2024-03-21T21:40:55.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 systemd[1]: Starting Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:40:55.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 podman[61457]: 2024-03-21 21:40:55.44379672 +0000 UTC m=+0.112610469 container create 94e44685fc2d91678a0ec90263d2fb08c30592495f55ac3ae0e3b880f13ba8d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:40:55.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 podman[61457]: 2024-03-21 21:40:55.349441902 +0000 UTC m=+0.018255602 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:40:55.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:55 smithi028 ceph-mon[29568]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-21T21:40:55.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:55 smithi028 ceph-mon[24806]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-21T21:40:56.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:55 smithi066 ceph-mon[25664]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-21T21:40:56.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 podman[61457]: 2024-03-21 21:40:55.637867636 +0000 UTC m=+0.306681330 container init 94e44685fc2d91678a0ec90263d2fb08c30592495f55ac3ae0e3b880f13ba8d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-21T21:40:56.002 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 podman[61457]: 2024-03-21 21:40:55.640538987 +0000 UTC m=+0.309352667 container start 94e44685fc2d91678a0ec90263d2fb08c30592495f55ac3ae0e3b880f13ba8d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-21T21:40:56.002 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 ceph-mgr[61476]: -- 172.21.15.66:0/2959850883 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bddf60a5a0 con 0x55bddf5df000 2024-03-21T21:40:56.002 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 bash[61457]: 94e44685fc2d91678a0ec90263d2fb08c30592495f55ac3ae0e3b880f13ba8d2 2024-03-21T21:40:56.002 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:55 smithi066 systemd[1]: Started Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:40:56.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:56 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:56.219+0000 7fb1bdd38200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:40:56.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:56 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:56.328+0000 7fb1bdd38200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:40:56.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:56 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:56.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:56 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:56.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:56 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:56.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:56 smithi028 ceph-mon[24806]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-21T21:40:56.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:56 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:56.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:56 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:56.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:56 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:56.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:56 smithi028 ceph-mon[29568]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-21T21:40:56.891 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:56 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:56.891 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:56 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:40:56.892 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:56 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:40:56.892 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:56 smithi066 ceph-mon[25664]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-21T21:40:57.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:57.037+0000 7fb1bdd38200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:40:57.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:57.139+0000 7fb1bdd38200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:40:57.619 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:57.291+0000 7fb1bdd38200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:40:57.620 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:57.392+0000 7fb1bdd38200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:40:57.620 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:57.481+0000 7fb1bdd38200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:40:57.887 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:57 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2019665522' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:40:57.887 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:57.761+0000 7fb1bdd38200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T21:40:57.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:57 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2019665522' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:40:57.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:57 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2019665522' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:40:58.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:57.886+0000 7fb1bdd38200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:40:58.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:57.984+0000 7fb1bdd38200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:40:58.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:58.094+0000 7fb1bdd38200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:40:58.565 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:58 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:58.278+0000 7fb1bdd38200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:40:59.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:40:59 smithi066 ceph-mon[25664]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:59.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:59 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:59.360+0000 7fb1bdd38200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:40:59.750 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:59 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:59.499+0000 7fb1bdd38200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:40:59.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:40:59 smithi028 ceph-mon[24806]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:40:59.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:40:59 smithi028 ceph-mon[29568]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:41:00.198 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:59 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:59.884+0000 7fb1bdd38200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:41:00.198 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:40:59 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:40:59.982+0000 7fb1bdd38200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:41:00.198 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:00 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:00.197+0000 7fb1bdd38200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:41:00.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:00 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:00.403+0000 7fb1bdd38200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:41:00.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:00 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:00.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:00 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:00.996 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:00 smithi066 ceph-mon[25664]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:41:00.997 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:00 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:00.714+0000 7fb1bdd38200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:41:00.997 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:00 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:00.995+0000 7fb1bdd38200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:41:01.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:41:00] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-03-21T21:41:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:00 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:00 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:00 smithi028 ceph-mon[29568]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:41:01.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:00 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:01.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:00 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:01.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:00 smithi028 ceph-mon[24806]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:41:01.665 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:01.254+0000 7fb1bdd38200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:41:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:02 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:02 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:02 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:02 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:02.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:02.142+0000 7fb1bdd38200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:41:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:02 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:02 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:02.923 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:02.749+0000 7fb1bdd38200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:41:02.923 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:02 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:02.854+0000 7fb1bdd38200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:41:03.216 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:03 smithi066 ceph-mon[25664]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:03.216 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:03.094+0000 7fb1bdd38200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:41:03.216 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: [21/Mar/2024:21:41:03] ENGINE Bus STARTING 2024-03-21T21:41:03.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:03 smithi028 ceph-mon[29568]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:03 smithi028 ceph-mon[24806]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:03.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: CherryPy Checker: 2024-03-21T21:41:03.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: The Application mounted at '' has an empty config. 2024-03-21T21:41:03.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:41:03.501 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: [21/Mar/2024:21:41:03] ENGINE Serving on http://:::9283 2024-03-21T21:41:03.501 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:41:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: [21/Mar/2024:21:41:03] ENGINE Bus STARTED 2024-03-21T21:41:04.359 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:04 smithi066 ceph-mon[25664]: Standby manager daemon x restarted 2024-03-21T21:41:04.359 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:04 smithi066 ceph-mon[25664]: Standby manager daemon x started 2024-03-21T21:41:04.359 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:04 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:41:04.359 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:04 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:41:04.359 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:04 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:41:04.359 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:04 smithi066 ceph-mon[25664]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:41:04.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[24806]: Standby manager daemon x restarted 2024-03-21T21:41:04.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[24806]: Standby manager daemon x started 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[24806]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[29568]: Standby manager daemon x restarted 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[29568]: Standby manager daemon x started 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:41:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:04 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/47986025' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: mgrmap e41: y(active, since 30s), standbys: x 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 393 B/s rd, 0 op/s 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 487 B/s rd, 0 op/s 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:41:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:41:05.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:05.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:41:05.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:05.286 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:05.286 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: mgrmap e41: y(active, since 30s), standbys: x 2024-03-21T21:41:05.286 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:05.286 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:05.286 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:05.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:41:05.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 393 B/s rd, 0 op/s 2024-03-21T21:41:05.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 487 B/s rd, 0 op/s 2024-03-21T21:41:05.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:05.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:41:05.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:41:05.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:05.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:41:05.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:05.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:05.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: mgrmap e41: y(active, since 30s), standbys: x 2024-03-21T21:41:05.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:05.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:05.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:05.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:41:05.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 393 B/s rd, 0 op/s 2024-03-21T21:41:05.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 487 B/s rd, 0 op/s 2024-03-21T21:41:05.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:05.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:41:05.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:41:05.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:05.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:41:05.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:06.911 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:41:07.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:07 smithi028 ceph-mon[29568]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:41:07.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:07 smithi028 ceph-mon[24806]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:41:07.626 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:41:07.626 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (13m) 27s ago 21m 23.2M - 0.23.0 ba2b418f427c cd065b97fad1 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 error 7s ago 20m - - 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (3m) 27s ago 20m 51.3M - 3.5 e1d6a67b021e 48135ddcc26c 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (12s) 7s ago 25m 68.0M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (10m) 27s ago 27m 474M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (27m) 27s ago 27m 75.3M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (26m) 7s ago 26m 55.4M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (26m) 27s ago 26m 51.2M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (13m) 27s ago 22m 21.1M - 1.3.1 1dbe0e931976 a8a424c65ebe 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (13m) 7s ago 21m 19.5M - 1.3.1 1dbe0e931976 f6cdb9cb9923 2024-03-21T21:41:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (25m) 27s ago 25m 53.5M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (25m) 27s ago 25m 54.3M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (24m) 27s ago 24m 51.8M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (24m) 27s ago 24m 52.4M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (24m) 7s ago 24m 55.1M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (23m) 7s ago 23m 52.4M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (23m) 7s ago 23m 51.8M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (22m) 7s ago 22m 52.7M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (17s) 7s ago 22m 51.4M - 2.33.4 514e6a882f6e fe8071f8b95e 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (20m) 27s ago 20m 76.5M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:41:07.628 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (20m) 7s ago 20m 74.9M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:41:07.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:07 smithi066 ceph-mon[25664]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:41:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:41:08.037 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:41:08.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:41:08.037 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:41:08.037 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T21:41:08.037 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:41:08.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:41:08.037 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:41:08.407 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:41:08.407 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:41:08.407 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:41:08.407 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-21T21:41:08.408 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-03-21T21:41:08.408 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr" 2024-03-21T21:41:08.408 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:41:08.408 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T21:41:08.408 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-21T21:41:08.408 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:41:08.408 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:41:08.213+0000 7f8a19e4f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:41:08.409 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:08.410 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return future.result(timeout) 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.__get_result() 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: raise self._exception 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:08.411 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:08.412 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:08.413 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:08.413 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:08.413 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:08.413 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:08.413 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:08.413 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:08.413 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:08.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2121494944' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:08.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2121494944' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:08 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:08.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:08.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2121494944' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:08.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:08.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:08 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:09.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:09.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:09.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: return future.result(timeout) 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.__get_result() 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: raise self._exception 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:09.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:09.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: Upgrade: Setting container_image for all mgr 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: Upgrade: Setting container_image for all crash 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: Upgrade: Setting container_image for all mds 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: Upgrade: Setting container_image for all nfs 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:41:09.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:09 smithi066 ceph-mon[25664]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:09.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:09.870 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:09.870 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:09.870 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:09.870 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:09.871 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: return future.result(timeout) 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.__get_result() 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: raise self._exception 2024-03-21T21:41:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:09.874 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:09.875 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: return future.result(timeout) 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:09.876 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.__get_result() 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: raise self._exception 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:09.877 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:09.878 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all mgr 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all crash 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all mds 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all nfs 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:41:09.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[24806]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all mgr 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all crash 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all mds 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all nfs 2024-03-21T21:41:09.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:41:09.881 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:41:09.881 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:09 smithi028 ceph-mon[29568]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:10.830 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:10 smithi028 systemd[1]: Stopping Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:41:11.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:41:10] "GET /metrics HTTP/1.1" 200 37978 "" "Prometheus/2.33.4" 2024-03-21T21:41:11.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:10 smithi028 podman[76478]: 2024-03-21 21:41:10.926621487 +0000 UTC m=+0.137341770 container died a8a424c65ebefab9d2f10fa68a210fadd3506b4518f36d50f2a34c1f041aea23 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:11.493 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:11 smithi028 podman[76478]: 2024-03-21 21:41:11.307687282 +0000 UTC m=+0.518407611 container cleanup a8a424c65ebefab9d2f10fa68a210fadd3506b4518f36d50f2a34c1f041aea23 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:11.493 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:11 smithi028 bash[76478]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a 2024-03-21T21:41:11.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:11 smithi066 ceph-mon[25664]: Upgrade: Updating node-exporter.a (1/2) 2024-03-21T21:41:11.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:11 smithi066 ceph-mon[25664]: Deploying daemon node-exporter.a on smithi028 2024-03-21T21:41:11.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:11 smithi066 ceph-mon[25664]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-03-21T21:41:11.919 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:11 smithi028 ceph-mon[29568]: Upgrade: Updating node-exporter.a (1/2) 2024-03-21T21:41:11.920 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:11 smithi028 ceph-mon[29568]: Deploying daemon node-exporter.a on smithi028 2024-03-21T21:41:11.920 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:11 smithi028 ceph-mon[29568]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-03-21T21:41:11.920 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:11 smithi028 podman[76492]: 2024-03-21 21:41:11.600638354 +0000 UTC m=+0.674823987 container remove a8a424c65ebefab9d2f10fa68a210fadd3506b4518f36d50f2a34c1f041aea23 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:11.920 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:11 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:41:11.920 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:11 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-21T21:41:11.920 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:11 smithi028 systemd[1]: Stopped Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:41:11.920 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:11 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 5.064s CPU time. 2024-03-21T21:41:11.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:11 smithi028 ceph-mon[24806]: Upgrade: Updating node-exporter.a (1/2) 2024-03-21T21:41:11.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:11 smithi028 ceph-mon[24806]: Deploying daemon node-exporter.a on smithi028 2024-03-21T21:41:11.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:11 smithi028 ceph-mon[24806]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-03-21T21:41:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:11 smithi028 systemd[1]: Starting Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:41:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:12 smithi028 bash[76622]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-21T21:41:12.946 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:12 smithi028 bash[76622]: Getting image source signatures 2024-03-21T21:41:12.946 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:12 smithi028 bash[76622]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-21T21:41:12.946 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:12 smithi028 bash[76622]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-21T21:41:12.947 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:12 smithi028 bash[76622]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-21T21:41:13.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:13 smithi066 ceph-mon[25664]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:41:13.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:13 smithi028 ceph-mon[24806]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:41:13.906 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:13 smithi028 ceph-mon[29568]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:41:14.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:13 smithi028 bash[76622]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-21T21:41:14.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 bash[76622]: Writing manifest to image destination 2024-03-21T21:41:14.696 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 podman[76622]: 2024-03-21 21:41:14.362163995 +0000 UTC m=+2.346576827 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-21T21:41:14.696 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 podman[76622]: 2024-03-21 21:41:14.460337468 +0000 UTC m=+2.444750292 container create 9ceebe17c0c78fcccf0452f69871f43144633877ef129cab6908df869571ad2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:15.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 podman[76622]: 2024-03-21 21:41:14.696013911 +0000 UTC m=+2.680426736 container init 9ceebe17c0c78fcccf0452f69871f43144633877ef129cab6908df869571ad2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:15.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 podman[76622]: 2024-03-21 21:41:14.699083691 +0000 UTC m=+2.683496518 container start 9ceebe17c0c78fcccf0452f69871f43144633877ef129cab6908df869571ad2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:15.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.703Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-21T21:41:15.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.703Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T21:41:15.198 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T21:41:15.199 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T21:41:15.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.704Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.705Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a[76674]: ts=2024-03-21T21:41:14.705Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 bash[76622]: 9ceebe17c0c78fcccf0452f69871f43144633877ef129cab6908df869571ad2b 2024-03-21T21:41:15.201 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:41:14 smithi028 systemd[1]: Started Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:41:15.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:15 smithi066 ceph-mon[25664]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T21:41:15.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:15 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:15.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:15 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:15 smithi028 ceph-mon[29568]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T21:41:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:15 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:15 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:15 smithi028 ceph-mon[24806]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T21:41:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:15 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:15 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:17.250 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:17 smithi066 systemd[1]: Stopping Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:41:17.696 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:17 smithi066 ceph-mon[25664]: Upgrade: Updating node-exporter.b (2/2) 2024-03-21T21:41:17.696 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:17 smithi066 ceph-mon[25664]: Deploying daemon node-exporter.b on smithi066 2024-03-21T21:41:17.696 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:17 smithi066 ceph-mon[25664]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:17.697 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:17 smithi066 podman[62818]: 2024-03-21 21:41:17.438120757 +0000 UTC m=+0.113581772 container died f6cdb9cb99234cd66daaffac9ebaf9ded30d1bd67908b7145b4c9f6736a9dcaf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:41:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:17 smithi028 ceph-mon[24806]: Upgrade: Updating node-exporter.b (2/2) 2024-03-21T21:41:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:17 smithi028 ceph-mon[24806]: Deploying daemon node-exporter.b on smithi066 2024-03-21T21:41:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:17 smithi028 ceph-mon[24806]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:17 smithi028 ceph-mon[29568]: Upgrade: Updating node-exporter.b (2/2) 2024-03-21T21:41:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:17 smithi028 ceph-mon[29568]: Deploying daemon node-exporter.b on smithi066 2024-03-21T21:41:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:17 smithi028 ceph-mon[29568]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:18.000 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:17 smithi066 podman[62818]: 2024-03-21 21:41:17.794302411 +0000 UTC m=+0.469763424 container cleanup f6cdb9cb99234cd66daaffac9ebaf9ded30d1bd67908b7145b4c9f6736a9dcaf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:41:18.000 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:17 smithi066 bash[62818]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b 2024-03-21T21:41:18.433 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:18 smithi066 podman[62832]: 2024-03-21 21:41:18.095795137 +0000 UTC m=+0.656429246 container remove f6cdb9cb99234cd66daaffac9ebaf9ded30d1bd67908b7145b4c9f6736a9dcaf (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:41:18.434 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:18 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:41:18.434 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:18 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-21T21:41:18.434 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:18 smithi066 systemd[1]: Stopped Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:41:18.434 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:18 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 5.532s CPU time. 2024-03-21T21:41:18.750 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:18 smithi066 systemd[1]: Starting Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:41:18.750 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:18 smithi066 bash[62964]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-21T21:41:19.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:19 smithi066 bash[62964]: Getting image source signatures 2024-03-21T21:41:19.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:19 smithi066 bash[62964]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-21T21:41:19.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:19 smithi066 bash[62964]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-21T21:41:19.500 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:19 smithi066 bash[62964]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-21T21:41:19.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:19 smithi028 ceph-mon[29568]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:41:19.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:19 smithi028 ceph-mon[24806]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:41:20.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:19 smithi066 ceph-mon[25664]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:41:20.784 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:20 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2667486539' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:41:20.784 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:20 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:20.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:20 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2667486539' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:41:20.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:20 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:20.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:20 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2667486539' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:41:20.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:20 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:21.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:41:20] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-03-21T21:41:21.250 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:20 smithi066 bash[62964]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-21T21:41:21.250 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:20 smithi066 bash[62964]: Writing manifest to image destination 2024-03-21T21:41:21.541 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-mon[25664]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:41:21.541 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 podman[62964]: 2024-03-21 21:41:21.215485955 +0000 UTC m=+2.682069595 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-21T21:41:21.541 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 podman[62964]: 2024-03-21 21:41:21.322550224 +0000 UTC m=+2.789133852 container create 93ce3d18115351dd6465735d23a3fabd52ccd870156165b0b2a272854882f37e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:41:21.804 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 podman[62964]: 2024-03-21 21:41:21.541072041 +0000 UTC m=+3.007655677 container init 93ce3d18115351dd6465735d23a3fabd52ccd870156165b0b2a272854882f37e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:41:21.804 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 podman[62964]: 2024-03-21 21:41:21.544672073 +0000 UTC m=+3.011255707 container start 93ce3d18115351dd6465735d23a3fabd52ccd870156165b0b2a272854882f37e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:41:21.804 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.549Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-21T21:41:21.804 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.549Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T21:41:21.805 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T21:41:21.806 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T21:41:21.807 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.550Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b[63015]: ts=2024-03-21T21:41:21.551Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 bash[62964]: 93ce3d18115351dd6465735d23a3fabd52ccd870156165b0b2a272854882f37e 2024-03-21T21:41:21.808 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:41:21 smithi066 systemd[1]: Started Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:41:21.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:21 smithi028 ceph-mon[24806]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:41:21.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:21 smithi028 ceph-mon[29568]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:41:22.808 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:22 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:22.808 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:22 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:22.808 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:22 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:22 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:22 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:22 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:22.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:22 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:22.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:22 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:22.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:22 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:23.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:23 smithi028 ceph-mon[24806]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:23.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:23 smithi028 ceph-mon[29568]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:24.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:23 smithi066 ceph-mon[25664]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:25.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:25 smithi028 ceph-mon[29568]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:41:25.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:25 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:25.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:25 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:25.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:25 smithi028 ceph-mon[24806]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:41:25.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:25 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:25.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:25 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:26.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:25 smithi066 ceph-mon[25664]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:41:26.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:25 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:26.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:25 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.610 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:27 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.611 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:27 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.611 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:27 smithi066 ceph-mon[25664]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:27.611 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:27 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.611 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:27 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[29568]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[24806]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:27 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:28.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:28 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:28.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:28 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:28 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:28 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:28.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:28 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:28 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:29.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:29 smithi028 ceph-mon[24806]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:29.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:29 smithi028 ceph-mon[29568]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:29.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:29 smithi066 ceph-mon[25664]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:30 smithi028 ceph-mon[24806]: from='client.25447 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:41:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:30 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:30 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:30.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:30 smithi028 ceph-mon[29568]: from='client.25447 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:41:30.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:30 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:30.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:30 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:30 smithi066 ceph-mon[25664]: from='client.25447 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:41:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:30 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:30 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:31.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:41:30] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-03-21T21:41:31.606 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[29568]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:31.606 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:31.606 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:31.606 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:31.606 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:41:31.606 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:31.606 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:41:31.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:31.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[24806]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:31.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:31.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:31.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:31.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:41:31.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:31.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:41:31.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:31 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:32.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:31 smithi066 ceph-mon[25664]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:41:32.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:31 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:32.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:31 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:32.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:31 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:32.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:31 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:41:32.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:31 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:32.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:31 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:41:32.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:31 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:32.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:32 smithi028 ceph-mon[24806]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:41:32.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:32 smithi028 ceph-mon[24806]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:41:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:32 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:41:32.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:32 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:41:32.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:32 smithi028 ceph-mon[29568]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:41:32.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:32 smithi028 ceph-mon[29568]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:41:32.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:32 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:41:32.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:32 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:41:33.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:32 smithi066 ceph-mon[25664]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:41:33.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:32 smithi066 ceph-mon[25664]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:41:33.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:32 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:41:33.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:32 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:41:34.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:41:34.343+0000 7f8a19e4f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:34.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:41:34.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:34.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:34.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:34.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:34.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:34.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return future.result(timeout) 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.__get_result() 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: raise self._exception 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:34.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:34.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:41:34.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:34.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:34.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:34.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:34.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:34.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:34.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:34.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:34.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:34.978 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:34.979 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:34 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:35.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:35 smithi066 ceph-mon[25664]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:35.751 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:35.752 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:35.753 INFO:journalctl@ceph.mon.b.smithi066.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: return future.result(timeout) 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.__get_result() 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: raise self._exception 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:35.754 INFO:journalctl@ceph.mon.b.smithi066.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:35.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:35.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:35.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:35.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:35.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:35.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:35.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:35.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.755 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:35.757 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:35.757 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.757 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.757 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:35.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:35 smithi028 ceph-mon[24806]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:35.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:35.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:35.947 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:35.948 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:35.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:35.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: return future.result(timeout) 2024-03-21T21:41:35.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:35.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.__get_result() 2024-03-21T21:41:35.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:35.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: raise self._exception 2024-03-21T21:41:35.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:35.950 INFO:journalctl@ceph.mon.a.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:35.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:35.953 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:35.953 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:35.953 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:35.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:35 smithi028 ceph-mon[29568]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:35.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:35.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:35.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:35.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:35.954 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:35.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:35.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:35.955 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:35.956 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:35.956 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:35.956 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:35.956 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:35.957 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:35.958 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:35.958 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:35.958 INFO:journalctl@ceph.mon.c.smithi028.stdout: return future.result(timeout) 2024-03-21T21:41:35.958 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:35.958 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.__get_result() 2024-03-21T21:41:35.958 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:35.958 INFO:journalctl@ceph.mon.c.smithi028.stdout: raise self._exception 2024-03-21T21:41:35.958 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.959 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.960 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:35.961 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:35.961 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:35.961 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:35.961 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:35.961 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:35.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:36.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:36 smithi066 ceph-mon[25664]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-03-21T21:41:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:36 smithi028 ceph-mon[24806]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-03-21T21:41:36.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:36 smithi028 ceph-mon[29568]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-03-21T21:41:37.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:37 smithi066 ceph-mon[25664]: Upgrade: Updating prometheus.a 2024-03-21T21:41:37.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:37 smithi066 ceph-mon[25664]: Deploying daemon prometheus.a on smithi066 2024-03-21T21:41:37.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:37 smithi028 ceph-mon[24806]: Upgrade: Updating prometheus.a 2024-03-21T21:41:37.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:37 smithi028 ceph-mon[24806]: Deploying daemon prometheus.a on smithi066 2024-03-21T21:41:37.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:37 smithi028 ceph-mon[29568]: Upgrade: Updating prometheus.a 2024-03-21T21:41:37.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:37 smithi028 ceph-mon[29568]: Deploying daemon prometheus.a on smithi066 2024-03-21T21:41:38.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:38 smithi028 ceph-mon[29568]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 1 op/s 2024-03-21T21:41:38.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:38 smithi028 ceph-mon[24806]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 1 op/s 2024-03-21T21:41:38.786 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:41:39.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:38 smithi066 ceph-mon[25664]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 1 op/s 2024-03-21T21:41:39.450 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:41:39.450 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (13m) 13s ago 21m 23.4M - 0.23.0 ba2b418f427c cd065b97fad1 2024-03-21T21:41:39.450 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 error 13s ago 21m - - 2024-03-21T21:41:39.450 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (20s) 13s ago 20m 45.9M - 3.5 e1d6a67b021e 036ff83a125a 2024-03-21T21:41:39.450 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (44s) 13s ago 26m 452M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:41:39.450 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (10m) 13s ago 28m 500M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (28m) 13s ago 28m 79.3M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (26m) 13s ago 26m 56.9M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (26m) 13s ago 26m 56.6M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (25s) 13s ago 22m 7990k - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (18s) 13s ago 22m 5435k - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (26m) 13s ago 26m 54.2M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (25m) 13s ago 25m 55.0M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (25m) 13s ago 25m 53.0M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (24m) 13s ago 24m 53.3M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (24m) 13s ago 24m 55.1M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (24m) 13s ago 24m 52.4M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (23m) 13s ago 23m 52.0M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (23m) 13s ago 23m 52.8M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (49s) 13s ago 22m 66.4M - 2.33.4 514e6a882f6e fe8071f8b95e 2024-03-21T21:41:39.451 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (21m) 13s ago 21m 76.8M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:41:39.452 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (20m) 13s ago 20m 75.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:41:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:41:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:41:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:41:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:41:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:41:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:41:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T21:41:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:41:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:41:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:41:39.905 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 systemd[1]: Stopping Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:41:40.181 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.904Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[60968]: ts=2024-03-21T21:41:39.905Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T21:41:40.182 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:39 smithi066 podman[64328]: 2024-03-21 21:41:39.915238869 +0000 UTC m=+0.132050295 container died fe8071f8b95eed5094f2316aef121a2c7f8717e66f9b0e8a319afc9bcade9195 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:40.201 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:41:40.201 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:41:40.201 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:41:40.201 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-21T21:41:40.201 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-03-21T21:41:40.202 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr" 2024-03-21T21:41:40.202 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:41:40.202 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T21:41:40.202 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-21T21:41:40.202 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:41:40.202 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:41:40.500 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:40 smithi066 podman[64328]: 2024-03-21 21:41:40.288111492 +0000 UTC m=+0.504922923 container cleanup fe8071f8b95eed5094f2316aef121a2c7f8717e66f9b0e8a319afc9bcade9195 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:40.500 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:40 smithi066 bash[64328]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T21:41:40.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:40 smithi066 ceph-mon[25664]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:40.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:40 smithi066 ceph-mon[25664]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 1 op/s 2024-03-21T21:41:40.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:40 smithi066 ceph-mon[25664]: from='client.25465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:40.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:40 smithi066 ceph-mon[25664]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:40.909 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:40 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2024991327' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:40.909 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:40 smithi066 podman[64343]: 2024-03-21 21:41:40.539153595 +0000 UTC m=+0.626501315 container remove fe8071f8b95eed5094f2316aef121a2c7f8717e66f9b0e8a319afc9bcade9195 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:40.910 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:40 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T21:41:40.910 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:40 smithi066 systemd[1]: Stopped Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:41:40.910 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:40 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.337s CPU time. 2024-03-21T21:41:40.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[29568]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:40.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[29568]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 1 op/s 2024-03-21T21:41:40.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[29568]: from='client.25465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:40.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[29568]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:40.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2024991327' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[24806]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[24806]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 1 op/s 2024-03-21T21:41:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[24806]: from='client.25465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[24806]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:40.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:40 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2024991327' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:41.235 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:40 smithi066 systemd[1]: Starting Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:41:41.235 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 podman[64476]: 2024-03-21 21:41:41.028008035 +0000 UTC m=+0.019795421 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-21T21:41:41.235 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 podman[64476]: 2024-03-21 21:41:41.15036025 +0000 UTC m=+0.142147625 container create f8d0d1c0130ba275b128708f377dcbf51f2e9fb9339c81cae57af1029eeb7288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:41.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-mon[25664]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:41.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:41.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:41.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:41.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 podman[64476]: 2024-03-21 21:41:41.327461845 +0000 UTC m=+0.319249229 container init f8d0d1c0130ba275b128708f377dcbf51f2e9fb9339c81cae57af1029eeb7288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:41.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 podman[64476]: 2024-03-21 21:41:41.330194287 +0000 UTC m=+0.321981664 container start f8d0d1c0130ba275b128708f377dcbf51f2e9fb9339c81cae57af1029eeb7288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:41.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.355Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-21T21:41:41.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.355Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-21T21:41:41.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.355Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi066 (none))" 2024-03-21T21:41:41.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.355Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T21:41:41.751 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.355Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.356Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.357Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.357Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.357Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.359Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.359Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.304µs 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.359Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.365Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.381Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.391Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.394Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.397Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-03-21T21:41:41.752 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.398Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.398Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.398Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=59.248µs wal_replay_duration=38.506996ms wbl_replay_duration=109ns total_replay_duration=38.867095ms 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.401Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.401Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.401Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.425Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.617071ms db_storage=863ns remote_storage=1.17µs web_handler=367ns query_engine=712ns scrape=411.747µs scrape_sd=89.089µs notify=10.341µs notify_sd=17.389µs rules=22.784952ms tracing=6.369µs 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.425Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:41:41.425Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 bash[64476]: f8d0d1c0130ba275b128708f377dcbf51f2e9fb9339c81cae57af1029eeb7288 2024-03-21T21:41:41.753 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:41:41 smithi066 systemd[1]: Started Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:41:41.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:41 smithi028 ceph-mon[24806]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:41 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:41 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:41 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:41.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:41 smithi028 ceph-mon[29568]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:41:41.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:41 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:41.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:41 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:41.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:41 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:42.589 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:42 smithi066 ceph-mon[25664]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-03-21T21:41:42.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:42 smithi028 ceph-mon[24806]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-03-21T21:41:42.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:42 smithi028 ceph-mon[29568]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-03-21T21:41:44.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:44 smithi028 ceph-mon[24806]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:41:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:44 smithi028 ceph-mon[29568]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:41:44.805 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:44 smithi066 ceph-mon[25664]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:41:46.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:46 smithi028 ceph-mon[24806]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:41:46.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:46 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:46.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:46 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:46.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:46 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2251109430' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:41:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:46 smithi028 ceph-mon[29568]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:41:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:46 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:46 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:46.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:46 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2251109430' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:41:46.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:46 smithi066 ceph-mon[25664]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:41:46.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:46 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:46.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:46 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:46.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:46 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2251109430' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:41:47.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:47 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:47.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:47 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:47.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:47 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:47.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:47 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:47.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:47 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:47.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:47 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:48.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:48 smithi066 ceph-mon[25664]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:48.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:48 smithi028 ceph-mon[24806]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:48 smithi028 ceph-mon[29568]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:41:50.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[24806]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:41:50.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:50.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:50.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:50.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:50.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:41:50.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:50.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:41:50.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:50.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[29568]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:41:50.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:50.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:50.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:50.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:50.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:41:50.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:50.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:41:50.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:51.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:50 smithi066 ceph-mon[25664]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:41:51.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:41:51.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:51.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:51.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:41:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:41:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:41:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:41:51.167 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:41:50] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.43.0" 2024-03-21T21:41:51.870 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:51 smithi028 ceph-mon[29568]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2024-03-21T21:41:51.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:51 smithi028 ceph-mon[29568]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 109 B/s rd, 0 op/s 2024-03-21T21:41:51.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:51 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:41:51.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:51 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:41:51.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:51 smithi028 ceph-mon[24806]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2024-03-21T21:41:51.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:51 smithi028 ceph-mon[24806]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 109 B/s rd, 0 op/s 2024-03-21T21:41:51.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:51 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:41:51.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:51 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:41:52.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:51 smithi066 ceph-mon[25664]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 0 op/s 2024-03-21T21:41:52.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:51 smithi066 ceph-mon[25664]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 109 B/s rd, 0 op/s 2024-03-21T21:41:52.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:51 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:41:52.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:51 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:41:53.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-03-21T21:41:53.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:41:53.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-03-21T21:41:53.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:41:53.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:53.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.544 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:41:53.331+0000 7f8a19e4f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:41:53.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:53.546 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return future.result(timeout) 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.__get_result() 2024-03-21T21:41:53.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: raise self._exception 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:53.548 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:53.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:53.550 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:53.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:53.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:53 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-03-21T21:41:54.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:41:54.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:53 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:41:54.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:54 smithi028 ceph-mon[29568]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:54.902 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:54.903 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: return future.result(timeout) 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.__get_result() 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: raise self._exception 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:54.904 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:54.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:54 smithi028 ceph-mon[24806]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:54.906 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:54.907 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:54.908 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:54.908 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:54.908 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:54.908 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:54.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:54 smithi028 ceph-mon[29568]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:41:54.909 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:54.909 INFO:journalctl@ceph.mon.a.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:54.909 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:54.909 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:54.909 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:54.909 INFO:journalctl@ceph.mon.a.smithi028.stdout: return future.result(timeout) 2024-03-21T21:41:54.909 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:54.909 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.__get_result() 2024-03-21T21:41:54.909 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: raise self._exception 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:54.910 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:54.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:54 smithi028 ceph-mon[24806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:54 smithi066 ceph-mon[25664]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:41:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:41:55.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: return future.result(timeout) 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.__get_result() 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: raise self._exception 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:41:55.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:41:55.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:41:55.005 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:41:55.005 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:41:55.005 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:41:55.005 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:41:55.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:54 smithi066 ceph-mon[25664]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T21:41:55.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:55 smithi028 ceph-mon[29568]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-03-21T21:41:55.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:55 smithi028 ceph-mon[24806]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-03-21T21:41:56.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:55 smithi066 ceph-mon[25664]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-03-21T21:41:56.807 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:56 smithi028 ceph-mon[29568]: Upgrade: Updating alertmanager.a 2024-03-21T21:41:56.807 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:56 smithi028 ceph-mon[29568]: Deploying daemon alertmanager.a on smithi028 2024-03-21T21:41:56.808 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:56 smithi028 ceph-mon[24806]: Upgrade: Updating alertmanager.a 2024-03-21T21:41:56.808 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:56 smithi028 ceph-mon[24806]: Deploying daemon alertmanager.a on smithi028 2024-03-21T21:41:57.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:56 smithi066 ceph-mon[25664]: Upgrade: Updating alertmanager.a 2024-03-21T21:41:57.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:56 smithi066 ceph-mon[25664]: Deploying daemon alertmanager.a on smithi028 2024-03-21T21:41:57.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:57 smithi028 ceph-mon[24806]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:41:57.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:57 smithi028 ceph-mon[29568]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:41:58.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:57 smithi066 ceph-mon[25664]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:41:58.866 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:58 smithi028 systemd[1]: Stopping Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:41:59.150 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:58 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[59310]: level=info ts=2024-03-21T21:41:58.865Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:41:59.151 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:58 smithi028 podman[79056]: 2024-03-21 21:41:58.906054907 +0000 UTC m=+0.153790759 container died cd065b97fad114063b61fc577e63d81c1e3d33853f5b83f2649fac9c95527555 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:59.446 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:59 smithi028 podman[79056]: 2024-03-21 21:41:59.323871927 +0000 UTC m=+0.571607777 container cleanup cd065b97fad114063b61fc577e63d81c1e3d33853f5b83f2649fac9c95527555 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:59.446 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:59 smithi028 bash[79056]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T21:41:59.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:41:59 smithi028 ceph-mon[29568]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:41:59.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:41:59 smithi028 ceph-mon[24806]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:41:59.836 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:59 smithi028 podman[79070]: 2024-03-21 21:41:59.541818225 +0000 UTC m=+0.631033735 container remove cd065b97fad114063b61fc577e63d81c1e3d33853f5b83f2649fac9c95527555 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:41:59.836 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:59 smithi028 podman[79070]: 2024-03-21 21:41:59.634231399 +0000 UTC m=+0.723446864 volume remove 2cbb7777266b2508439c73a6b713a217a836ddb2c635b4cbd93b43bbf6323e3f 2024-03-21T21:42:00.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:41:59 smithi066 ceph-mon[25664]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:42:00.196 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:59 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T21:42:00.196 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:59 smithi028 systemd[1]: Stopped Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:42:00.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:41:59 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.549s CPU time. 2024-03-21T21:42:00.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 systemd[1]: Starting Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:42:00.588 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 podman[79202]: 2024-03-21 21:42:00.164641546 +0000 UTC m=+0.017927227 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-21T21:42:00.588 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 podman[79202]: 2024-03-21 21:42:00.294314269 +0000 UTC m=+0.147599992 volume create 5539a45423ea6137983ee7a206cf1a7f2f8c43543067faebd8a4bda26b8cde6e 2024-03-21T21:42:00.588 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 podman[79202]: 2024-03-21 21:42:00.386338477 +0000 UTC m=+0.239624155 container create a09458e22230b7cbf566694f9004cc2c136baf339f6f835dfb16bed05184ea03 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:42:00.864 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:42:00] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.43.0" 2024-03-21T21:42:01.196 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 podman[79202]: 2024-03-21 21:42:00.864061872 +0000 UTC m=+0.717347551 container init a09458e22230b7cbf566694f9004cc2c136baf339f6f835dfb16bed05184ea03 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 podman[79202]: 2024-03-21 21:42:00.866822027 +0000 UTC m=+0.720107713 container start a09458e22230b7cbf566694f9004cc2c136baf339f6f835dfb16bed05184ea03 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:00.879Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:00.879Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:00.880Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.28 port=9094 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:00.882Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:00.905Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:00.905Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:00.908Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:00.908Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 bash[79202]: a09458e22230b7cbf566694f9004cc2c136baf339f6f835dfb16bed05184ea03 2024-03-21T21:42:01.197 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:00 smithi028 systemd[1]: Started Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:42:01.910 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:01 smithi066 ceph-mon[25664]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T21:42:01.911 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:01 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:01.911 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:01 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:01.911 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:01 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:01.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:01 smithi028 ceph-mon[29568]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T21:42:01.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:01 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:01.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:01 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:01.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:01 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:01.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:01 smithi028 ceph-mon[24806]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T21:42:01.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:01 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:01.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:01 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:01.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:01 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:02.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:02 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:02.882Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000071495s 2024-03-21T21:42:03.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:03 smithi028 ceph-mon[29568]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:42:03.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:03 smithi028 ceph-mon[24806]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:42:04.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:03 smithi066 ceph-mon[25664]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:42:05.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:05 smithi028 ceph-mon[24806]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:42:05.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:05.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:05.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:05 smithi028 ceph-mon[29568]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:42:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:06.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:05 smithi066 ceph-mon[25664]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:42:06.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:06.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:06.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:06.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:06 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:06.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:06 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:06.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:06 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1733160573' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:42:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:06 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:06 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:06 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1733160573' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:42:07.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:06 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:07.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:06 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:07.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:06 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1733160573' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:42:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:07 smithi028 ceph-mon[29568]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:42:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:07 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:07 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:07.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:07 smithi028 ceph-mon[24806]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:42:07.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:07 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:07.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:07 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:08.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:07 smithi066 ceph-mon[25664]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T21:42:08.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:07 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:08.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:07 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:09.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:09 smithi028 ceph-mon[24806]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:42:09.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:09 smithi028 ceph-mon[29568]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:42:10.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:09 smithi066 ceph-mon[25664]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:42:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:42:11.138 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:42:10] "GET /metrics HTTP/1.1" 200 37971 "" "Prometheus/2.43.0" 2024-03-21T21:42:11.138 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:42:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:42:10.883Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00136152s 2024-03-21T21:42:11.258 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:42:11.258 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (11s) 5s ago 22m 16.4M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:42:11.258 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 error 6s ago 22m - - 2024-03-21T21:42:11.258 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 unknown 5s ago 21m - - 2024-03-21T21:42:11.258 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (75s) 6s ago 26m 452M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:42:11.258 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (11m) 5s ago 28m 504M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:42:11.258 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (28m) 5s ago 28m 82.0M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (27m) 6s ago 27m 55.6M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (27m) 5s ago 27m 58.1M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (56s) 5s ago 23m 11.6M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (50s) 6s ago 23m 14.0M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (26m) 5s ago 26m 54.1M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (26m) 5s ago 26m 55.0M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (25m) 5s ago 25m 53.0M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (25m) 5s ago 25m 53.3M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (25m) 6s ago 25m 55.1M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (24m) 6s ago 24m 52.4M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (24m) 6s ago 24m 52.1M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (23m) 6s ago 23m 52.8M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:42:11.259 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (30s) 6s ago 23m 53.7M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:42:11.260 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (21m) 5s ago 21m 77.0M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:42:11.260 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (21m) 6s ago 21m 75.5M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:42:11.672 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:42:11.672 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:42:11.672 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:42:11.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:42:11.672 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:42:11.673 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:42:11.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:42:11.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:11.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:11.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:11.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:11.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2024-03-21T21:42:11.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 124 B/s rd, 0 op/s 2024-03-21T21:42:11.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:11.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[29568]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2024-03-21T21:42:11.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 124 B/s rd, 0 op/s 2024-03-21T21:42:11.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:11.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:42:11.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:42:11.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:11.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:42:11.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:11.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:11 smithi028 ceph-mon[24806]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 100 B/s rd, 0 op/s 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 124 B/s rd, 0 op/s 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:42:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:42:11.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:11.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:42:11.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:11.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:11 smithi066 ceph-mon[25664]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:42:12.017 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:42:12.017 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:42:12.018 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:42:12.018 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-21T21:42:12.018 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-03-21T21:42:12.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr" 2024-03-21T21:42:12.018 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:42:12.018 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T21:42:12.018 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading alertmanager daemons", 2024-03-21T21:42:12.018 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:42:12.018 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:42:12.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:12 smithi028 ceph-mon[24806]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:12.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:12 smithi028 ceph-mon[24806]: from='client.25549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:12.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:12 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3524131235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:12.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:12 smithi028 ceph-mon[29568]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:12.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:12 smithi028 ceph-mon[29568]: from='client.25549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:12.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:12 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3524131235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:12.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:12 smithi066 ceph-mon[25664]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:12.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:12 smithi066 ceph-mon[25664]: from='client.25549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:12.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:12 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3524131235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:13.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-mon[29568]: from='client.25561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:13.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-mon[29568]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-03-21T21:42:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-mon[24806]: from='client.25561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-mon[24806]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-03-21T21:42:13.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:13 smithi066 ceph-mon[25664]: from='client.25561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:13.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:13 smithi066 ceph-mon[25664]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-03-21T21:42:14.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:42:13.962+0000 7f8a19e4f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:14.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:42:14.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:14.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:14.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:14.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:14.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:14.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:42:14.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:14.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return future.result(timeout) 2024-03-21T21:42:14.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.__get_result() 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: raise self._exception 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: config_blobs=daemon_spec.final_config, 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:14.200 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:14.201 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.833 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:42:14.833 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:14.833 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.833 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.833 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.833 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.833 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.833 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.834 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.834 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.834 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.834 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.834 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:14 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:14.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:14 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:15.537 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:15 smithi066 ceph-mon[25664]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:15.537 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:15.537 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:15.537 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:15.537 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:15.537 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.537 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.537 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:42:15.537 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.538 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: return future.result(timeout) 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.__get_result() 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: raise self._exception 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:42:15.539 INFO:journalctl@ceph.mon.b.smithi066.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:42:15.540 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:15 smithi066 ceph-mon[25664]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:42:15.541 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:15 smithi066 ceph-mon[25664]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 623 B/s rd, 0 op/s 2024-03-21T21:42:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:15 smithi028 ceph-mon[29568]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:42:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: return future.result(timeout) 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.__get_result() 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: raise self._exception 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:15.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:15 smithi028 ceph-mon[24806]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:42:15.951 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:42:15.952 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:15 smithi028 ceph-mon[29568]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:15 smithi028 ceph-mon[29568]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 623 B/s rd, 0 op/s 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.a.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:42:15.953 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: return future.result(timeout) 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.__get_result() 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: raise self._exception 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:15.954 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:15.955 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:15.956 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:15.956 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:15.956 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:15.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:15 smithi028 ceph-mon[24806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T21:42:15.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:15 smithi028 ceph-mon[24806]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 623 B/s rd, 0 op/s 2024-03-21T21:42:16.598 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:16 smithi066 ceph-mon[25664]: Upgrade: Updating grafana.a 2024-03-21T21:42:16.598 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:16 smithi066 ceph-mon[25664]: Deploying daemon grafana.a on smithi066 2024-03-21T21:42:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:16 smithi028 ceph-mon[24806]: Upgrade: Updating grafana.a 2024-03-21T21:42:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:16 smithi028 ceph-mon[24806]: Deploying daemon grafana.a on smithi066 2024-03-21T21:42:16.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:16 smithi028 ceph-mon[29568]: Upgrade: Updating grafana.a 2024-03-21T21:42:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:16 smithi028 ceph-mon[29568]: Deploying daemon grafana.a on smithi066 2024-03-21T21:42:17.536 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:17 smithi066 ceph-mon[25664]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-21T21:42:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:17 smithi028 ceph-mon[24806]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-21T21:42:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:17 smithi028 ceph-mon[29568]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-21T21:42:19.000 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:18 smithi066 systemd[1]: Starting Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:42:19.500 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 podman[66442]: 2024-03-21 21:42:18.990564834 +0000 UTC m=+0.019041507 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-21T21:42:19.500 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 podman[66442]: 2024-03-21 21:42:19.109970843 +0000 UTC m=+0.138447518 container create c4f0c2d28ad98f188e4d730ed924c87e6830db4e452846ac632b3923da8b288a (image=quay.io/ceph/grafana:9.4.12, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T21:42:19.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-mon[25664]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-21T21:42:19.753 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 podman[66442]: 2024-03-21 21:42:19.529337527 +0000 UTC m=+0.557814214 container init c4f0c2d28ad98f188e4d730ed924c87e6830db4e452846ac632b3923da8b288a (image=quay.io/ceph/grafana:9.4.12, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T21:42:19.753 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 podman[66442]: 2024-03-21 21:42:19.532576078 +0000 UTC m=+0.561052756 container start c4f0c2d28ad98f188e4d730ed924c87e6830db4e452846ac632b3923da8b288a (image=quay.io/ceph/grafana:9.4.12, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T21:42:19.753 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 bash[66442]: c4f0c2d28ad98f188e4d730ed924c87e6830db4e452846ac632b3923da8b288a 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 systemd[1]: Started Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689144624Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.68933218Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689347603Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689352632Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689356789Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689361079Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689364874Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689368851Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689372921Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689377517Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689381634Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689385524Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-21T21:42:19.754 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689389613Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-21T21:42:19.755 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689394553Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-21T21:42:19.755 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689398498Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-21T21:42:19.755 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689403192Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-21T21:42:19.755 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689406933Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-21T21:42:19.755 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.689410998Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-21T21:42:19.755 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=settings t=2024-03-21T21:42:19.68941503Z level=info msg="App mode production" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=sqlstore t=2024-03-21T21:42:19.68947147Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=sqlstore t=2024-03-21T21:42:19.689488872Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.70172542Z level=info msg="Starting DB migrations" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.705432521Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.716959922Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.718119308Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.718917899Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.720209664Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.721357601Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.722467049Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.723255737Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.723986639Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.724559374Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.725483461Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-21T21:42:19.756 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.726695179Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.727077248Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.72826499Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.72940192Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.730547214Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.731542731Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.732802011Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.733920987Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.734780639Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.735894061Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.737013516Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.737938027Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.738768641Z level=info msg="Executing migration" id="create alert_image table" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.739616567Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.740478637Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.741310252Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-21T21:42:19.757 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.74215981Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-21T21:42:19.758 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.744877666Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T21:42:19.758 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.744968629Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T21:42:19.758 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.747789946Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-21T21:42:19.758 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.750364089Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-21T21:42:19.758 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.753109637Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-21T21:42:19.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:19 smithi028 ceph-mon[24806]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-21T21:42:19.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:19 smithi028 ceph-mon[29568]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-21T21:42:20.252 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.755915086Z level=info msg="Executing migration" id="create secrets table" 2024-03-21T21:42:20.252 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.758644697Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-21T21:42:20.252 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.76520885Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-21T21:42:20.252 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.76802517Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-21T21:42:20.252 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.770596172Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-21T21:42:20.252 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.77780802Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.784438562Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.7872382Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.789933946Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.792777526Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.795507962Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.798286125Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.80099042Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.803732067Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.806401289Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.809118743Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.811805226Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.814468968Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.817097932Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-21T21:42:20.253 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.819892133Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.822601309Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.825394292Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.828094275Z level=info msg="Executing migration" id="add correlation config column" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.83081022Z level=info msg="Executing migration" id="create entity_events table" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.833488443Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.836106372Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.836201425Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.838925825Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.839047141Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.841459637Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.84196179Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.842397954Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-21T21:42:20.254 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.844952612Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.847615216Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.850247841Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.852940409Z level=info msg="Executing migration" id="Drop public config table" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.853698765Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.854521257Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.855310926Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.856081168Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.856919614Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.860865975Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.862040624Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.863218244Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-21T21:42:20.255 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.863560405Z level=info msg="Executing migration" id="add share column" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.864826746Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.865612195Z level=info msg="Executing migration" id="create file table" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.866075637Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.866557662Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.86704273Z level=info msg="Executing migration" id="create file_meta table" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.867470657Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.86794696Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.868269598Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.8693908Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.870169676Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.870818398Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.872070598Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-21T21:42:20.256 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.87240455Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.872886827Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.873308652Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.873859716Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.874261133Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.875519043Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.876751617Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.877567421Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.891095953Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.89187963Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.892351652Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.896848117Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.897372264Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.897887971Z level=info msg="Executing migration" id="create folder table" 2024-03-21T21:42:20.257 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.898309201Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-21T21:42:20.258 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.898787055Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-21T21:42:20.258 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.899264025Z level=info msg="Executing migration" id="Update folder title length" 2024-03-21T21:42:20.258 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.899541155Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-21T21:42:20.258 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=migrator t=2024-03-21T21:42:19.900034874Z level=info msg="migrations completed" performed=105 skipped=376 duration=194.637993ms 2024-03-21T21:42:20.258 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=sqlstore t=2024-03-21T21:42:19.900367982Z level=info msg="Created default organization" 2024-03-21T21:42:20.259 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=plugin.finder t=2024-03-21T21:42:19.934096809Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-21T21:42:20.259 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=secrets t=2024-03-21T21:42:19.934366169Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-21T21:42:20.259 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=query_data t=2024-03-21T21:42:19.937661024Z level=info msg="Query Service initialization" 2024-03-21T21:42:20.259 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=live.push_http t=2024-03-21T21:42:19.940057426Z level=info msg="Live Push Gateway initialization" 2024-03-21T21:42:20.851 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:20.851 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:20.851 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:20.851 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=infra.usagestats.collector t=2024-03-21T21:42:20.606150379Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=provisioning.datasources t=2024-03-21T21:42:20.608010994Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=provisioning.datasources t=2024-03-21T21:42:20.608020187Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=provisioning.datasources t=2024-03-21T21:42:20.608119242Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=provisioning.datasources t=2024-03-21T21:42:20.619427591Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=provisioning.alerting t=2024-03-21T21:42:20.630291794Z level=info msg="starting to provision alerting" 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=provisioning.alerting t=2024-03-21T21:42:20.630304321Z level=info msg="finished to provision alerting" 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=ngalert.state.manager t=2024-03-21T21:42:20.631446159Z level=info msg="Warming state cache for startup" 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=grafanaStorageLogger t=2024-03-21T21:42:20.630929122Z level=info msg="storage starting" 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=http.server t=2024-03-21T21:42:20.640892416Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=ngalert.state.manager t=2024-03-21T21:42:20.650263899Z level=info msg="State cache has been initialized" states=0 duration=18.816742ms 2024-03-21T21:42:20.852 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=ticker t=2024-03-21T21:42:20.650316519Z level=info msg=starting first_tick=2024-03-21T21:42:30Z 2024-03-21T21:42:20.853 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=ngalert.multiorg.alertmanager t=2024-03-21T21:42:20.650331229Z level=info msg="starting MultiOrg Alertmanager" 2024-03-21T21:42:20.853 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=sqlstore.transactions t=2024-03-21T21:42:20.65856574Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T21:42:20.853 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=sqlstore.transactions t=2024-03-21T21:42:20.669634361Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-21T21:42:20.853 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:42:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=sqlstore.transactions t=2024-03-21T21:42:20.681315379Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-21T21:42:20.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:42:20] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.43.0" 2024-03-21T21:42:20.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:20 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:20.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:20 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:20.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:20 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:20.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:20 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:20.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:20 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:20.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:20 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:20.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:20 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:20.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:20 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:21.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:21 smithi066 ceph-mon[25664]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T21:42:21.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:21 smithi028 ceph-mon[24806]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T21:42:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:21 smithi028 ceph-mon[29568]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T21:42:23.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:23 smithi028 ceph-mon[24806]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:42:23.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:23 smithi028 ceph-mon[29568]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:42:24.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:23 smithi066 ceph-mon[25664]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T21:42:25.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:25 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:25.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:25 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:25.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:25 smithi028 ceph-mon[29568]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:42:25.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:25 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:25.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:25 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:25.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:25 smithi028 ceph-mon[24806]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:42:25.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:25 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:25.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:25 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:25.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:25 smithi066 ceph-mon[25664]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T21:42:26.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:26 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:26.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:26 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:26.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:26 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1666309744' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:42:26.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:26 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:26.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:26 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:26.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:26 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1666309744' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:42:26.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:26 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:26.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:26 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:26.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:26 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1666309744' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:42:27.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:27 smithi066 ceph-mon[25664]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:42:27.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:27 smithi028 ceph-mon[29568]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:42:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:27 smithi028 ceph-mon[24806]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T21:42:29.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[24806]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[29568]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:29.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:29.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:29.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:29.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:29.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:42:29.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:29 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:29 smithi066 ceph-mon[25664]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-21T21:42:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:29 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:29 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:29 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:29 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:30.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:29 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:30.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:29 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:42:30.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:29 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:30.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[24806]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2024-03-21T21:42:30.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[24806]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2024-03-21T21:42:30.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:42:30.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:42:30.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[24806]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:42:30.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[29568]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2024-03-21T21:42:30.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[29568]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2024-03-21T21:42:30.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:42:30.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:42:30.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-mon[29568]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:42:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:30 smithi066 ceph-mon[25664]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 98 B/s rd, 0 op/s 2024-03-21T21:42:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:30 smithi066 ceph-mon[25664]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2024-03-21T21:42:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:30 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:42:31.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:30 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:42:31.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:30 smithi066 ceph-mon[25664]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T21:42:31.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:42:30] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.43.0" 2024-03-21T21:42:32.513 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:42:32.357+0000 7f8a19e4f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:32.514 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:42:32.515 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:32.515 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:32.515 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:32.515 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:32.515 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:32.515 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:32.515 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:32.516 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:32.516 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:32.516 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:42:32.516 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:42:32.516 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:42:32.516 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:42:32.516 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:42:32.516 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:42:32.516 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:42:32.518 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:42:32.518 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return future.result(timeout) 2024-03-21T21:42:32.518 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:42:32.518 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return self.__get_result() 2024-03-21T21:42:32.518 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:42:32.519 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: raise self._exception 2024-03-21T21:42:32.519 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:42:32.519 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: config_blobs=daemon_spec.final_config, 2024-03-21T21:42:32.519 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:42:32.519 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:42:32.519 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Creating ceph-iscsi config... 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: Traceback (most recent call last): 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: return _run_code(code, main_globals, None, 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:32.521 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: exec(code, run_globals) 2024-03-21T21:42:32.522 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:32.522 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:32.522 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:32.522 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:32.522 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:32.522 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:32.522 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:32.522 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:32.522 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:42:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:42:32.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:42:32.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:42:32.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:42:32.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:42:32.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:42:32.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:42:32.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:42:32.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:32.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:42:32.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:32.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:32.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:32 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:42:33.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:42:33.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:33.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:32 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:33.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:33.765 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:33.766 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:42:33.767 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: return future.result(timeout) 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.__get_result() 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: raise self._exception 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:33.768 INFO:journalctl@ceph.mon.a.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:33.769 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:42:33.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:42:33.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: return future.result(timeout) 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.__get_result() 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: raise self._exception 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:33.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:33.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: Upgrade: Finalizing container_image settings 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: Upgrade: Complete! 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:42:33.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:42:33.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:42:33.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:33.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[24806]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:42:33.775 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:33.775 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:33.775 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:33.775 INFO:journalctl@ceph.mon.c.smithi028.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:33.775 INFO:journalctl@ceph.mon.c.smithi028.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:33.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:42:33.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: Upgrade: Finalizing container_image settings 2024-03-21T21:42:33.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: Upgrade: Complete! 2024-03-21T21:42:33.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:42:33.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:42:33.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:42:33.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:42:33.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:33.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:33 smithi028 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:42:34.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:34.001 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T21:42:34.002 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: return future.result(timeout) 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: return self.__get_result() 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: raise self._exception 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: Creating ceph-iscsi config... 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: Non-zero exit code 1 from systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:34.003 INFO:journalctl@ceph.mon.b.smithi066.stdout: systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: Traceback (most recent call last): 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: exec(code, run_globals) 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:42:34.004 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1123, in deploy_daemon 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout: RuntimeError: Failed command: systemctl restart ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout: See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: Upgrade: Finalizing container_image settings 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: Upgrade: Complete! 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: Reconfiguring iscsi.foo.smithi028.lrtsfq (dependencies changed)... 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:34.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:33 smithi066 ceph-mon[25664]: Reconfiguring daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:42:35.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[29568]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:42:35.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:35.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:35.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[24806]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:42:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:42:35.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:35 smithi066 ceph-mon[25664]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:42:36.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:42:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:42:36.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:35 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:42:36.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:36 smithi028 ceph-mon[29568]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:42:36.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:36 smithi028 ceph-mon[29568]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:42:36.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:36 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/2487046029' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:42:36.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:36 smithi028 ceph-mon[24806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:42:36.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:36 smithi028 ceph-mon[24806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:42:36.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:36 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/2487046029' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:42:37.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:36 smithi066 ceph-mon[25664]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T21:42:37.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:36 smithi066 ceph-mon[25664]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T21:42:37.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:36 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/2487046029' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:42:37.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:37 smithi028 ceph-mon[29568]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 1 op/s 2024-03-21T21:42:37.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:37 smithi028 ceph-mon[24806]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 1 op/s 2024-03-21T21:42:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:37 smithi066 ceph-mon[25664]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 1 op/s 2024-03-21T21:42:39.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:39 smithi028 ceph-mon[24806]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-03-21T21:42:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:39 smithi028 ceph-mon[29568]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-03-21T21:42:40.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:39 smithi066 ceph-mon[25664]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-03-21T21:42:40.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:42:40] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.43.0" 2024-03-21T21:42:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[24806]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 230 B/s rd, 0 op/s 2024-03-21T21:42:40.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:40.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:40.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:40.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:40.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:40.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[29568]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 230 B/s rd, 0 op/s 2024-03-21T21:42:40.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:40 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:41.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:40 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:40 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:40 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:42:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:40 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:42:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:40 smithi066 ceph-mon[25664]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 230 B/s rd, 0 op/s 2024-03-21T21:42:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:40 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:41.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:41 smithi028 ceph-mon[24806]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:42:41.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:41 smithi028 ceph-mon[24806]: Cluster is now healthy 2024-03-21T21:42:41.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:41 smithi028 ceph-mon[29568]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:42:41.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:41 smithi028 ceph-mon[29568]: Cluster is now healthy 2024-03-21T21:42:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:41 smithi066 ceph-mon[25664]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T21:42:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:41 smithi066 ceph-mon[25664]: Cluster is now healthy 2024-03-21T21:42:43.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:42 smithi066 ceph-mon[25664]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 774 B/s rd, 0 op/s 2024-03-21T21:42:43.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:42 smithi028 ceph-mon[29568]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 774 B/s rd, 0 op/s 2024-03-21T21:42:43.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:42 smithi028 ceph-mon[24806]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 774 B/s rd, 0 op/s 2024-03-21T21:42:43.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-21T21:42:43.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:43 smithi028 ceph-mon[24806]: from='client.25615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:43.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:43 smithi028 ceph-mon[29568]: from='client.25615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:44.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:43 smithi066 ceph-mon[25664]: from='client.25615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:44.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:44 smithi028 ceph-mon[24806]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-03-21T21:42:44.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:44 smithi028 ceph-mon[29568]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-03-21T21:42:45.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:44 smithi066 ceph-mon[25664]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-03-21T21:42:45.244 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:42:45.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:45 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1277171686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:45.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:45 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1277171686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:45.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-21T21:42:46.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:45 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/1277171686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:46.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:46 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:42:46.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:46 smithi028 ceph-mon[29568]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-03-21T21:42:46.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:46 smithi028 ceph-mon[24806]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:42:46.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:46 smithi028 ceph-mon[24806]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-03-21T21:42:47.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:46 smithi066 ceph-mon[25664]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:42:47.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:46 smithi066 ceph-mon[25664]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-03-21T21:42:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)" 2024-03-21T21:42:48.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-03-21T21:42:48.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:48 smithi028 ceph-mon[29568]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-21T21:42:48.840 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:48 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3127423084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:48.840 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:48 smithi028 ceph-mon[24806]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-21T21:42:48.840 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:48 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3127423084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:49.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:48 smithi066 ceph-mon[25664]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-21T21:42:49.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:48 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3127423084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:50.601 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:42:50.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:42:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:42:50] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.43.0" 2024-03-21T21:42:50.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:50 smithi028 ceph-mon[24806]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-21T21:42:50.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:50.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:50.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:50 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/970173532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:50.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:50 smithi028 ceph-mon[29568]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-21T21:42:50.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:50.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:50.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:50 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/970173532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:51.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:50 smithi066 ceph-mon[25664]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-21T21:42:51.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:42:51.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:50 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:42:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:50 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/970173532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:42:51.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-03-21T21:42:52.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:52 smithi028 ceph-mon[29568]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:42:52.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:52 smithi028 ceph-mon[24806]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:42:53.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:52 smithi066 ceph-mon[25664]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:42:54.581 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:54 smithi028 ceph-mon[29568]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:54.582 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:54 smithi028 ceph-mon[29568]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:42:54.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:54 smithi028 ceph-mon[24806]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:54.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:54 smithi028 ceph-mon[24806]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:42:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:54 smithi066 ceph-mon[25664]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:42:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:54 smithi066 ceph-mon[25664]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:42:55.382 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:42:56.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade status' 2024-03-21T21:42:56.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:56 smithi028 ceph-mon[24806]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:42:56.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:56 smithi028 ceph-mon[24806]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:42:56.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:56 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:42:56.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:56 smithi028 ceph-mon[29568]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:42:57.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:56 smithi066 ceph-mon[25664]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:42:57.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:56 smithi066 ceph-mon[25664]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:42:57.998 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:42:57.998 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": null, 2024-03-21T21:42:57.998 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": false, 2024-03-21T21:42:57.998 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "", 2024-03-21T21:42:57.998 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:42:57.998 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": null, 2024-03-21T21:42:57.998 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-03-21T21:42:57.998 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:42:57.998 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:42:58.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:42:59.049 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:58 smithi028 ceph-mon[29568]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:42:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:58 smithi028 ceph-mon[24806]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:42:59.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:58 smithi066 ceph-mon[25664]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:00.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:59 smithi028 ceph-mon[24806]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:00.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:42:59 smithi028 ceph-mon[24806]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:00.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:59 smithi028 ceph-mon[29568]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:00.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:42:59 smithi028 ceph-mon[29568]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:00.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:59 smithi066 ceph-mon[25664]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:00.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:42:59 smithi066 ceph-mon[25664]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:00.984 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-03-21T21:43:01.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:01 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/4108896863' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:43:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:01 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/4108896863' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:43:01.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:43:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:43:00] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.43.0" 2024-03-21T21:43:01.415 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:01 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/4108896863' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:43:01.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-03-21T21:43:02.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:02 smithi028 ceph-mon[29568]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:02.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:02 smithi028 ceph-mon[24806]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:02.361 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:02 smithi066 ceph-mon[25664]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:04.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:04 smithi028 ceph-mon[24806]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:04.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:04 smithi028 ceph-mon[24806]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:04 smithi028 ceph-mon[29568]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:04 smithi028 ceph-mon[29568]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:05.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:04 smithi066 ceph-mon[25664]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:05.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:04 smithi066 ceph-mon[25664]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:05.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:05 smithi028 ceph-mon[24806]: from='client.25663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "mon", "hosts": "smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:05.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:05 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:05 smithi028 ceph-mon[29568]: from='client.25663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "mon", "hosts": "smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:05 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:06.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:05 smithi066 ceph-mon[25664]: from='client.25663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "mon", "hosts": "smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:06.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:05 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:06.226 INFO:teuthology.orchestra.run.smithi028.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:06.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[24806]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:06.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[24806]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:06.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:06.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:06.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:06.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:06.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:06.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:06.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[29568]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:06.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:06.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:06.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:06.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:06.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:06 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:06.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T21:43:07.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:06 smithi066 ceph-mon[25664]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:07.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:06 smithi066 ceph-mon[25664]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:06 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:06 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:06 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:06 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:06 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:07.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:07 smithi028 ceph-mon[24806]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:07.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:07 smithi028 ceph-mon[24806]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:07 smithi028 ceph-mon[29568]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:07 smithi028 ceph-mon[29568]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:08.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:07 smithi066 ceph-mon[25664]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:08.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:07 smithi066 ceph-mon[25664]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:08.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:08 smithi028 ceph-mon[29568]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:08 smithi028 ceph-mon[24806]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:09.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:08 smithi066 ceph-mon[25664]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all mgr 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: Upgrade: It appears safe to stop mon.b 2024-03-21T21:43:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[24806]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all mgr 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: Upgrade: It appears safe to stop mon.b 2024-03-21T21:43:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:09 smithi028 ceph-mon[29568]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:10.084 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:43:10.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: Upgrade: Setting container_image for all mgr 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: Upgrade: It appears safe to stop mon.b 2024-03-21T21:43:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:09 smithi066 ceph-mon[25664]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (70s) 31s ago 23m 20.0M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (51s) 46s ago 23m 53.4M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (35s) 31s ago 22m 47.9M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (2m) 46s ago 27m 452M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (12m) 31s ago 29m 512M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (29m) 31s ago 29m 83.4M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (28m) 46s ago 28m 56.4M 2048M 17.2.0 e1d6a67b021e 67a113787b9a 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (28m) 31s ago 28m 58.4M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (116s) 31s ago 24m 15.4M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:43:10.760 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (109s) 46s ago 24m 14.4M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (27m) 31s ago 27m 54.2M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (27m) 31s ago 27m 55.0M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (26m) 31s ago 26m 53.0M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (26m) 31s ago 26m 53.4M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (26m) 46s ago 26m 55.1M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (25m) 46s ago 25m 52.4M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (25m) 46s ago 25m 52.1M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (24m) 46s ago 24m 52.8M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (89s) 46s ago 24m 58.4M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (22m) 31s ago 22m 77.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:43:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (22m) 46s ago 22m 75.6M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:43:11.173 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:43:11.173 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:43:11.173 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T21:43:11.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:43:11.173 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:43:11.174 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:43:11.175 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:43:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:43:10] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.43.0" 2024-03-21T21:43:11.513 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:43:11.513 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:43:11.513 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:43:11.513 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi066", 2024-03-21T21:43:11.513 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:43:11.513 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "0/1 daemons upgraded", 2024-03-21T21:43:11.514 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading mon daemons", 2024-03-21T21:43:11.514 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:43:11.514 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: from='client.15774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: from='client.25675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: Upgrade: Updating mon.b 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: Deploying daemon mon.b on smithi066 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:11.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:11 smithi066 ceph-mon[25664]: from='client.? 172.21.15.28:0/3407503747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:11.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: from='client.15774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:11.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: from='client.25675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:11.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: Upgrade: Updating mon.b 2024-03-21T21:43:11.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:11.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:43:11.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: Deploying daemon mon.b on smithi066 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3407503747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: from='client.15774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: from='client.25675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: Upgrade: Updating mon.b 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: Deploying daemon mon.b on smithi066 2024-03-21T21:43:11.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:11.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:11 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3407503747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:12.916 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:12 smithi066 ceph-mon[25664]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:12.916 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:12 smithi066 ceph-mon[25664]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:12.916 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:12 smithi066 systemd[1]: Stopping Ceph mon.b for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:43:12.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:12 smithi028 ceph-mon[24806]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:12 smithi028 ceph-mon[24806]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:12.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:12 smithi028 ceph-mon[29568]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:12.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:12 smithi028 ceph-mon[29568]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:13.226 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b[25655]: 2024-03-21T21:43:12.913+0000 7f9d11439700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:43:13.227 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b[25655]: 2024-03-21T21:43:12.913+0000 7f9d11439700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-21T21:43:13.227 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:12 smithi066 podman[67916]: 2024-03-21 21:43:12.955330872 +0000 UTC m=+0.141385206 container died 67a113787b9a6106d8296cba15844852e09b0b399bac27aebf890fd1e52944ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b, maintainer=Guillaume Abrioux , ceph=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.openshift.expose-services=, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-21T21:43:13.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:13 smithi066 podman[67916]: 2024-03-21 21:43:13.314001156 +0000 UTC m=+0.500055489 container cleanup 67a113787b9a6106d8296cba15844852e09b0b399bac27aebf890fd1e52944ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b, version=8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.component=centos-stream-container, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_CLEAN=True, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True, vcs-type=git, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-21T21:43:13.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:13 smithi066 bash[67916]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b 2024-03-21T21:43:13.926 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:13 smithi066 podman[67930]: 2024-03-21 21:43:13.598674803 +0000 UTC m=+0.636706557 container remove 67a113787b9a6106d8296cba15844852e09b0b399bac27aebf890fd1e52944ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, release=754, RELEASE=HEAD, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, version=8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-21T21:43:13.926 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:13 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-21T21:43:13.926 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:13 smithi066 systemd[1]: Stopped Ceph mon.b for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:43:13.926 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:13 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.b.service: Consumed 18.854s CPU time. 2024-03-21T21:43:14.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:13 smithi066 systemd[1]: Starting Ceph mon.b for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:43:14.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 podman[68063]: 2024-03-21 21:43:14.046675389 +0000 UTC m=+0.018498664 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:43:14.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 podman[68063]: 2024-03-21 21:43:14.149664203 +0000 UTC m=+0.121487509 container create ad84920145372514f1edb5021ce037c84a6bc5872da1b7b14e19cd76954d9dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-21T21:43:14.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 podman[68063]: 2024-03-21 21:43:14.385266544 +0000 UTC m=+0.357089821 container init ad84920145372514f1edb5021ce037c84a6bc5872da1b7b14e19cd76954d9dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:43:14.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 podman[68063]: 2024-03-21 21:43:14.388315783 +0000 UTC m=+0.360139048 container start ad84920145372514f1edb5021ce037c84a6bc5872da1b7b14e19cd76954d9dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-21T21:43:14.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T21:43:14.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev), process ceph-mon, pid 2 2024-03-21T21:43:14.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: pidfile_write: ignore empty --pid-file 2024-03-21T21:43:14.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: load: jerasure load: lrc 2024-03-21T21:43:14.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: RocksDB version: 7.9.2 2024-03-21T21:43:14.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Git sha 0 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Compile date 2024-03-21 16:40:47 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: DB SUMMARY 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: DB Session ID: K8WKBOWIDNXDHSAAAPAZ 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: CURRENT file: CURRENT 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2814 Bytes 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 3524238 ; 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.error_if_exists: 0 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.create_if_missing: 0 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.paranoid_checks: 1 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T21:43:14.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.env: 0x55e089360b20 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.info_log: 0x55e08b4a3920 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.statistics: (nil) 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.use_fsync: 0 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_log_file_size: 0 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.allow_fallocate: 1 2024-03-21T21:43:14.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.use_direct_reads: 0 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.db_log_dir: 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.wal_dir: 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T21:43:14.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.write_buffer_manager: 0x55e08b4741e0 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.unordered_write: 0 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T21:43:14.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T21:43:14.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T21:43:14.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T21:43:14.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.row_cache: None 2024-03-21T21:43:14.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.wal_filter: None 2024-03-21T21:43:14.757 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.two_write_queues: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.wal_compression: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.atomic_flush: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.log_readahead_size: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T21:43:14.758 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_background_jobs: 2 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_background_compactions: -1 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_subcompactions: 1 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T21:43:14.759 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_open_files: -1 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_background_flushes: -1 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Compression algorithms supported: 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: kZSTD supported: 0 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: kXpressCompression supported: 0 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T21:43:14.760 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: kZlibCompression supported: 1 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: kSnappyCompression supported: 1 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: kLZ4Compression supported: 1 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: kBZip2Compression supported: 0 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.merge_operator: 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_filter: None 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T21:43:14.761 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e08b4a3a20) 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: cache_index_and_filter_blocks: 1 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: pin_top_level_index_and_filter: 1 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: index_type: 0 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: data_block_index_type: 0 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: index_shortening: 1 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: checksum: 4 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: no_block_cache: 0 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: block_cache: 0x55e08b4a7090 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: block_cache_name: BinnedLRUCache 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: block_cache_options: 2024-03-21T21:43:14.762 INFO:journalctl@ceph.mon.b.smithi066.stdout: capacity : 536870912 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: num_shard_bits : 4 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: strict_capacity_limit : 0 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: high_pri_pool_ratio: 0.000 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: block_cache_compressed: (nil) 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: persistent_cache: (nil) 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: block_size: 4096 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: block_size_deviation: 10 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: block_restart_interval: 16 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: index_block_restart_interval: 1 2024-03-21T21:43:14.763 INFO:journalctl@ceph.mon.b.smithi066.stdout: metadata_block_size: 4096 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: partition_filters: 0 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: use_delta_encoding: 1 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: filter_policy: bloomfilter 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: whole_key_filtering: 1 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: verify_compression: 0 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: read_amp_bytes_per_bit: 0 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: format_version: 5 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: enable_index_compression: 1 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: block_align: 0 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: max_auto_readahead_size: 262144 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: prepopulate_block_cache: 0 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: initial_auto_readahead_size: 8192 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T21:43:14.764 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression: NoCompression 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.num_levels: 7 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T21:43:14.765 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T21:43:14.766 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T21:43:14.767 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T21:43:14.768 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.table_properties_collectors: 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.inplace_update_support: 0 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T21:43:14.769 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.bloom_locality: 0 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.max_successive_merges: 0 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.ttl: 2592000 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T21:43:14.770 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.enable_blob_files: false 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.min_blob_size: 0 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28861, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T21:43:14.771 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a6be8e75-ea45-4570-bb97-c6569a233de3 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711057394418069, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711057394427008, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 2138230, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28866, "largest_seqno": 30801, "table_properties": {"data_size": 2130604, "index_size": 4750, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1861, "raw_key_size": 17599, "raw_average_key_size": 23, "raw_value_size": 2114882, "raw_average_value_size": 2850, "num_data_blocks": 214, "num_entries": 742, "num_filter_entries": 742, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711057394, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a6be8e75-ea45-4570-bb97-c6569a233de3", "db_session_id": "K8WKBOWIDNXDHSAAAPAZ", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711057394427112, "job": 1, "event": "recovery_finished"} 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e08b596000 2024-03-21T21:43:14.772 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: DB pointer 0x55e08b580000 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout: ** DB Stats ** 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T21:43:14.773 INFO:journalctl@ceph.mon.b.smithi066.stdout: 2024-03-21T21:43:14.774 INFO:journalctl@ceph.mon.b.smithi066.stdout: ** Compaction Stats [default] ** 2024-03-21T21:43:14.774 INFO:journalctl@ceph.mon.b.smithi066.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T21:43:14.774 INFO:journalctl@ceph.mon.b.smithi066.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T21:43:14.774 INFO:journalctl@ceph.mon.b.smithi066.stdout: L0 1/0 2.04 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 555.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-21T21:43:14.774 INFO:journalctl@ceph.mon.b.smithi066.stdout: L6 1/0 8.37 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-21T21:43:14.774 INFO:journalctl@ceph.mon.b.smithi066.stdout: Sum 2/0 10.41 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 555.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-21T21:43:14.774 INFO:journalctl@ceph.mon.b.smithi066.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 555.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-21T21:43:14.774 INFO:journalctl@ceph.mon.b.smithi066.stdout: 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: ** Compaction Stats [default] ** 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 555.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T21:43:14.775 INFO:journalctl@ceph.mon.b.smithi066.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T21:43:14.776 INFO:journalctl@ceph.mon.b.smithi066.stdout: Cumulative compaction: 0.00 GB write, 179.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:43:14.776 INFO:journalctl@ceph.mon.b.smithi066.stdout: Interval compaction: 0.00 GB write, 179.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:43:14.776 INFO:journalctl@ceph.mon.b.smithi066.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T21:43:14.776 INFO:journalctl@ceph.mon.b.smithi066.stdout: Block cache BinnedLRUCache@0x55e08b4a7090#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout: 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: starting mon.b rank 2 at public addrs [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] at bind addrs [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: mon.b@-1(???) e3 preinit fsid c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: mon.b@-1(???).mds e1 new map 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: mon.b@-1(???).mds e1 print_map 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout: e1 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout: legacy client fscid: -1 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout: 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout: No filesystems configured 2024-03-21T21:43:14.777 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: mon.b@-1(???).osd e112 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-21T21:43:14.778 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: mon.b@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T21:43:14.778 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: mon.b@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T21:43:14.778 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: mon.b@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T21:43:14.778 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 ceph-mon[68082]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-03-21T21:43:14.778 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 bash[68063]: ad84920145372514f1edb5021ce037c84a6bc5872da1b7b14e19cd76954d9dd6 2024-03-21T21:43:14.778 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:14 smithi066 systemd[1]: Started Ceph mon.b for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:43:15.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: mon.b calling monitor election 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: mon.c calling monitor election 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: mon.a calling monitor election 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: fsmap 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: mgrmap e41: y(active, since 2m), standbys: x 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: overall HEALTH_OK 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:15.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:15 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: mon.b calling monitor election 2024-03-21T21:43:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: mon.c calling monitor election 2024-03-21T21:43:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: mon.a calling monitor election 2024-03-21T21:43:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:43:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} 2024-03-21T21:43:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: fsmap 2024-03-21T21:43:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: mgrmap e41: y(active, since 2m), standbys: x 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: overall HEALTH_OK 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: mon.b calling monitor election 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: mon.c calling monitor election 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: mon.a calling monitor election 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} 2024-03-21T21:43:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: fsmap 2024-03-21T21:43:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T21:43:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: mgrmap e41: y(active, since 2m), standbys: x 2024-03-21T21:43:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: overall HEALTH_OK 2024-03-21T21:43:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:15.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:15 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:16.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:16 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:16.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:16 smithi066 ceph-mon[68082]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:16 smithi028 ceph-mon[24806]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:16 smithi028 ceph-mon[24806]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:16.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:16 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:16.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:16 smithi028 ceph-mon[29568]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:18.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:17 smithi028 ceph-mon[24806]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:18.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:17 smithi028 ceph-mon[29568]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:18.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:17 smithi066 ceph-mon[68082]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:19.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:43:19 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:43:19.435+0000 7f8a1ce55700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-21T21:43:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:20 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:20 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:20 smithi028 ceph-mon[24806]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:20 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:20 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:20 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:20.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:20 smithi028 ceph-mon[29568]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:20.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:20 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:20.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:20 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:20.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:20 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:20.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:20 smithi066 ceph-mon[68082]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:20.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:20 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:21.171 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:43:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:43:20] "GET /metrics HTTP/1.1" 200 37990 "" "Prometheus/2.43.0" 2024-03-21T21:43:21.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:21 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:21.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:21 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:21.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:21 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:21.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:21 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:21.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:21 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:21.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:21 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:22.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:22 smithi028 ceph-mon[29568]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:22.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:22 smithi028 ceph-mon[24806]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:22.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:22 smithi066 ceph-mon[68082]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:25.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: Detected new or changed devices on smithi066 2024-03-21T21:43:25.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:25.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:25.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:25.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:43:25.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:43:25.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:43:25.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:43:25.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:43:25.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:43:25.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:43:25.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:24 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: Detected new or changed devices on smithi066 2024-03-21T21:43:25.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:25.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:25.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-21T21:43:25.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: Detected new or changed devices on smithi066 2024-03-21T21:43:25.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:43:25.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:43:25.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:43:25.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:25.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:24 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all crash 2024-03-21T21:43:25.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all mds 2024-03-21T21:43:25.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:43:25.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:43:25.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:43:25.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all nfs 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all node-exporter 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all prometheus 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all alertmanager 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all grafana 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all loki 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all promtail 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Finalizing container_image settings 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: Upgrade: Complete! 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all crash 2024-03-21T21:43:25.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all mds 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all nfs 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all node-exporter 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all prometheus 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all alertmanager 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all grafana 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all loki 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all promtail 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Finalizing container_image settings 2024-03-21T21:43:25.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: Upgrade: Complete! 2024-03-21T21:43:25.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:25 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:26.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all crash 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mds 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nfs 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all node-exporter 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all prometheus 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all alertmanager 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all grafana 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all loki 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all promtail 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Finalizing container_image settings 2024-03-21T21:43:26.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: Upgrade: Complete! 2024-03-21T21:43:26.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:25 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:27.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:26 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:27.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:26 smithi066 ceph-mon[68082]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:27.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:26 smithi028 ceph-mon[24806]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:27.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:26 smithi028 ceph-mon[24806]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:27.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:26 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:27.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:26 smithi028 ceph-mon[29568]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:29.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:28 smithi066 ceph-mon[68082]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:28 smithi028 ceph-mon[29568]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:29.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:28 smithi028 ceph-mon[24806]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:30 smithi066 ceph-mon[68082]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:31.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:30 smithi028 ceph-mon[29568]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:31.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:30 smithi028 ceph-mon[24806]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:43:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:43:30] "GET /metrics HTTP/1.1" 200 37990 "" "Prometheus/2.43.0" 2024-03-21T21:43:32.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:32 smithi028 ceph-mon[29568]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:32.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:32 smithi028 ceph-mon[24806]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:33.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:32 smithi066 ceph-mon[68082]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:34.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:34 smithi028 ceph-mon[24806]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:34.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:34 smithi028 ceph-mon[29568]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:35.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:34 smithi066 ceph-mon[68082]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:35.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:35 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:35.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:35 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:36.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:35 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:36.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:36 smithi028 ceph-mon[24806]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:36.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:36 smithi028 ceph-mon[24806]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:36.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:36 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:36.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:36 smithi028 ceph-mon[29568]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:37.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:36 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:37.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:36 smithi066 ceph-mon[68082]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:38.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:37 smithi028 ceph-mon[24806]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:38.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:37 smithi028 ceph-mon[29568]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:38.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:37 smithi066 ceph-mon[68082]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:40.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:39 smithi028 ceph-mon[24806]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:40.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:39 smithi028 ceph-mon[29568]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:40.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:39 smithi066 ceph-mon[68082]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:41.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:43:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:43:40] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.43.0" 2024-03-21T21:43:42.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:41 smithi028 ceph-mon[29568]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:42.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:41 smithi028 ceph-mon[24806]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:42.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:41 smithi066 ceph-mon[68082]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:42.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:43:43.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:42 smithi028 ceph-mon[29568]: from='client.25699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:43.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:42 smithi028 ceph-mon[24806]: from='client.25699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:43.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:42 smithi066 ceph-mon[68082]: from='client.25699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:44.472 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (104s) 64s ago 24m 20.0M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (85s) 25s ago 23m 79.0M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (69s) 64s ago 22m 47.9M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (2m) 25s ago 28m 453M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (13m) 64s ago 30m 512M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (30m) 64s ago 30m 83.4M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (30s) 25s ago 28m 19.1M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (28m) 64s ago 28m 58.4M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (2m) 64s ago 24m 15.4M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (2m) 25s ago 24m 14.2M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (28m) 64s ago 28m 54.2M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:43:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (27m) 64s ago 27m 55.0M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:43:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (27m) 64s ago 27m 53.0M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:43:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (26m) 64s ago 26m 53.4M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:43:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (26m) 25s ago 26m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:43:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (26m) 25s ago 26m 52.7M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:43:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (25m) 25s ago 25m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:43:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (25m) 25s ago 25m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:43:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (2m) 25s ago 24m 64.8M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:43:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (23m) 64s ago 23m 77.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:43:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (23m) 25s ago 23m 76.0M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:43:44.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:44 smithi028 ceph-mon[29568]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:44.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:44 smithi028 ceph-mon[24806]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:45.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:44 smithi066 ceph-mon[68082]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:45.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-03-21T21:43:45.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:45 smithi028 ceph-mon[29568]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:45.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:45 smithi028 ceph-mon[24806]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:46.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:45 smithi066 ceph-mon[68082]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:46 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:46 smithi028 ceph-mon[29568]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:46.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:46 smithi028 ceph-mon[24806]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:46.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:46 smithi028 ceph-mon[24806]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:47.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:46 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:47.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:46 smithi066 ceph-mon[68082]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:47.237 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:43:47.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:47 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/1719181537' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:47.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:47 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/1719181537' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:47.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade status' 2024-03-21T21:43:48.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:47 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/1719181537' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:43:48.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:48 smithi028 ceph-mon[24806]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:48 smithi028 ceph-mon[29568]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:49.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:48 smithi066 ceph-mon[68082]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:49.859 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:43:49.860 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": null, 2024-03-21T21:43:49.860 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": false, 2024-03-21T21:43:49.860 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "", 2024-03-21T21:43:49.860 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:43:49.860 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": null, 2024-03-21T21:43:49.861 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-03-21T21:43:49.861 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:43:49.861 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:43:50.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:49 smithi028 ceph-mon[29568]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:50.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:49 smithi028 ceph-mon[24806]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:50.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:49 smithi066 ceph-mon[68082]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:50.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:43:51.043 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:43:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:43:50] "GET /metrics HTTP/1.1" 200 37924 "" "Prometheus/2.43.0" 2024-03-21T21:43:51.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:50 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:51.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:50 smithi028 ceph-mon[29568]: from='client.25717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:51.044 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:50 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:51.044 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:50 smithi028 ceph-mon[24806]: from='client.25717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:51.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:50 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:43:51.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:50 smithi066 ceph-mon[68082]: from='client.25717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:51 smithi028 ceph-mon[29568]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:52.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:51 smithi028 ceph-mon[24806]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:52.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:51 smithi066 ceph-mon[68082]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:52.506 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-03-21T21:43:53.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:52 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/3353147477' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:43:53.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:52 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/3353147477' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:43:53.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-03-21T21:43:53.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:52 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/3353147477' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:43:54.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:54 smithi028 ceph-mon[24806]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:54.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:54 smithi028 ceph-mon[29568]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:54 smithi066 ceph-mon[68082]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:56.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:55 smithi028 ceph-mon[29568]: from='client.25729 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:56.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:55 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:56.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:55 smithi028 ceph-mon[29568]: from='client.25735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "mon", "hosts": "smithi028", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:56.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:55 smithi028 ceph-mon[29568]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:56.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:55 smithi028 ceph-mon[24806]: from='client.25729 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:56.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:55 smithi028 ceph-mon[24806]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:56.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:55 smithi028 ceph-mon[24806]: from='client.25735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "mon", "hosts": "smithi028", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:56.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:55 smithi028 ceph-mon[24806]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:56.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:55 smithi066 ceph-mon[68082]: from='client.25729 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:56.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:55 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:43:56.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:55 smithi066 ceph-mon[68082]: from='client.25735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "mon", "hosts": "smithi028", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:43:56.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:55 smithi066 ceph-mon[68082]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:43:57.605 INFO:teuthology.orchestra.run.smithi028.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:58.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T21:43:58.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[24806]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:58.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:58.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:58.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:58.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:58.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:58.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[24806]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:58.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[29568]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:58.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:58.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:58.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:58.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:58.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:58.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:58 smithi028 ceph-mon[29568]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:59.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:58 smithi066 ceph-mon[68082]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:59.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:58 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:59.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:58 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:43:59.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:58 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:43:59.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:58 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:43:59.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:58 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:43:59.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:58 smithi066 ceph-mon[68082]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:43:59.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:43:59 smithi028 ceph-mon[29568]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:43:59.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:43:59 smithi028 ceph-mon[24806]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:44:00.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:43:59 smithi066 ceph-mon[68082]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:44:00.491 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:44:00.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:00 smithi028 ceph-mon[29568]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:00.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:00 smithi028 ceph-mon[24806]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:00.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:44:00] "GET /metrics HTTP/1.1" 200 37924 "" "Prometheus/2.43.0" 2024-03-21T21:44:01.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:00 smithi066 ceph-mon[68082]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:01.176 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (2m) 81s ago 24m 20.0M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (102s) 42s ago 23m 79.0M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (86s) 81s ago 23m 47.9M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (3m) 42s ago 28m 453M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (13m) 81s ago 30m 512M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (30m) 81s ago 30m 83.4M 2048M 17.2.0 e1d6a67b021e 355fc6af68b0 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (47s) 42s ago 29m 19.1M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (29m) 81s ago 29m 58.4M 2048M 17.2.0 e1d6a67b021e 953f34022a7f 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (2m) 81s ago 24m 15.4M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (2m) 42s ago 24m 14.2M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (28m) 81s ago 28m 54.2M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (28m) 81s ago 28m 55.0M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:44:01.177 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (27m) 81s ago 27m 53.0M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:44:01.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (27m) 81s ago 27m 53.4M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:44:01.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (26m) 42s ago 26m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:44:01.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (26m) 42s ago 26m 52.7M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:44:01.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (26m) 42s ago 26m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:44:01.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (25m) 42s ago 25m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:44:01.178 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (2m) 42s ago 24m 64.8M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:44:01.178 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (23m) 81s ago 23m 77.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:44:01.178 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (23m) 42s ago 23m 76.0M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:44:01.588 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 1 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": {}, 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:44:01.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:44:01.590 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:44:01.590 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-21T21:44:01.590 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:44:01.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:44:01.590 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:44:01.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[29568]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:01.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:44:01.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:01.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[29568]: from='client.? 172.21.15.28:0/572257276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[24806]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-21T21:44:01.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:01 smithi028 ceph-mon[24806]: from='client.? 172.21.15.28:0/572257276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:44:01.949 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:44:01.949 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:44:01.949 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:44:01.949 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi028", 2024-03-21T21:44:01.949 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:44:01.949 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "0/2 daemons upgraded", 2024-03-21T21:44:01.949 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading mon daemons", 2024-03-21T21:44:01.950 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:44:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:44:02.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:01 smithi066 ceph-mon[68082]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:02.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:01 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:44:02.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:01 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:02.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:01 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:44:02.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:01 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:44:02.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:01 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T21:44:02.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:01 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-21T21:44:02.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:01 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/572257276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:44:02.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[24806]: from='client.25747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[24806]: from='client.15858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[24806]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[24806]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[24806]: Upgrade: Setting container_image for all mgr 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[24806]: Upgrade: It appears safe to stop mon.a 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[24806]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[29568]: from='client.25747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[29568]: from='client.15858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[29568]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:44:02.843 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[29568]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:44:02.844 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[29568]: Upgrade: Setting container_image for all mgr 2024-03-21T21:44:02.844 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[29568]: Upgrade: It appears safe to stop mon.a 2024-03-21T21:44:02.844 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:02 smithi028 ceph-mon[29568]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:02.902 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:02 smithi066 ceph-mon[68082]: from='client.25747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:02.902 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:02 smithi066 ceph-mon[68082]: from='client.15858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:02.902 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:02 smithi066 ceph-mon[68082]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:44:02.902 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:02 smithi066 ceph-mon[68082]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:44:02.902 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:02 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mgr 2024-03-21T21:44:02.903 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:02 smithi066 ceph-mon[68082]: Upgrade: It appears safe to stop mon.a 2024-03-21T21:44:02.903 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:02 smithi066 ceph-mon[68082]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:03.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[24806]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:03.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:44:03.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:03.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:03.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[24806]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:03.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[29568]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:03.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:44:03.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:03.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:03.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:03 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:04.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:03 smithi066 ceph-mon[68082]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:04.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:03 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' 2024-03-21T21:44:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:03 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:03 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:03 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:04.751 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:04 smithi028 ceph-mon[29568]: Upgrade: Updating mon.a 2024-03-21T21:44:04.751 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:04 smithi028 ceph-mon[29568]: Deploying daemon mon.a on smithi028 2024-03-21T21:44:04.751 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:04 smithi028 ceph-mon[29568]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:04.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:04 smithi028 ceph-mon[24806]: Upgrade: Updating mon.a 2024-03-21T21:44:04.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:04 smithi028 ceph-mon[24806]: Deploying daemon mon.a on smithi028 2024-03-21T21:44:04.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:04 smithi028 ceph-mon[24806]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:05.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:04 smithi066 ceph-mon[68082]: Upgrade: Updating mon.a 2024-03-21T21:44:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:04 smithi066 ceph-mon[68082]: Deploying daemon mon.a on smithi028 2024-03-21T21:44:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:04 smithi066 ceph-mon[68082]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:05.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:04 smithi028 systemd[1]: Stopping Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:44:05.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:05 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a[24802]: 2024-03-21T21:44:05.004+0000 7fb2fbeca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:44:05.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:05 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a[24802]: 2024-03-21T21:44:05.004+0000 7fb2fbeca700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-21T21:44:05.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:05 smithi028 podman[85629]: 2024-03-21 21:44:05.056431786 +0000 UTC m=+0.160578563 container died 355fc6af68b078ea4037f61d19a370f967492c1558bd826df4b2525cb699cbdf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, distribution-scope=public, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-type=git, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, version=8) 2024-03-21T21:44:05.340 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:05 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:05] ENGINE Bus STOPPING 2024-03-21T21:44:05.341 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:05 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:05] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:44:05.341 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:05 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:05] ENGINE Bus STOPPED 2024-03-21T21:44:05.341 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:05 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:05] ENGINE Bus STARTING 2024-03-21T21:44:05.341 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:05 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:05] ENGINE Serving on http://:::9283 2024-03-21T21:44:05.341 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:05 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:05] ENGINE Bus STARTED 2024-03-21T21:44:05.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:05 smithi028 podman[85629]: 2024-03-21 21:44:05.44630759 +0000 UTC m=+0.550454356 container cleanup 355fc6af68b078ea4037f61d19a370f967492c1558bd826df4b2525cb699cbdf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, name=centos-stream, distribution-scope=public, ceph=True) 2024-03-21T21:44:05.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:05 smithi028 bash[85629]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a 2024-03-21T21:44:05.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:05 smithi028 podman[85655]: 2024-03-21 21:44:05.689321716 +0000 UTC m=+0.631437558 container remove 355fc6af68b078ea4037f61d19a370f967492c1558bd826df4b2525cb699cbdf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.openshift.expose-services=, distribution-scope=public, vcs-type=git, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, release=754, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream) 2024-03-21T21:44:05.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:05 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-21T21:44:05.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:05 smithi028 systemd[1]: Stopped Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:44:05.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:05 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.a.service: Consumed 26.688s CPU time. 2024-03-21T21:44:06.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 systemd[1]: Starting Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:44:06.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 podman[85785]: 2024-03-21 21:44:06.14153698 +0000 UTC m=+0.018361476 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:44:06.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 podman[85785]: 2024-03-21 21:44:06.257427549 +0000 UTC m=+0.134252095 container create 66c02b062f9ea5faeb7d0c05ca86383330b9fc3ff9559f3a18d86a2a8edd2c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:44:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 podman[85785]: 2024-03-21 21:44:06.460068784 +0000 UTC m=+0.336893285 container init 66c02b062f9ea5faeb7d0c05ca86383330b9fc3ff9559f3a18d86a2a8edd2c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0) 2024-03-21T21:44:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 podman[85785]: 2024-03-21 21:44:06.463133169 +0000 UTC m=+0.339957668 container start 66c02b062f9ea5faeb7d0c05ca86383330b9fc3ff9559f3a18d86a2a8edd2c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-21T21:44:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T21:44:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev), process ceph-mon, pid 2 2024-03-21T21:44:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: pidfile_write: ignore empty --pid-file 2024-03-21T21:44:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: load: jerasure load: lrc 2024-03-21T21:44:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: RocksDB version: 7.9.2 2024-03-21T21:44:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Git sha 0 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Compile date 2024-03-21 16:40:47 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: DB SUMMARY 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: DB Session ID: SQ7X2O3OV03381V1GNT9 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: CURRENT file: CURRENT 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: MANIFEST file: MANIFEST-000015 size: 3080 Bytes 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000063.sst 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000061.log size: 459420 ; 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.error_if_exists: 0 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.create_if_missing: 0 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.paranoid_checks: 1 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T21:44:06.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T21:44:06.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T21:44:06.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.env: 0x55e44545ab20 2024-03-21T21:44:06.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T21:44:06.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.info_log: 0x55e44629f920 2024-03-21T21:44:06.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T21:44:06.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.statistics: (nil) 2024-03-21T21:44:06.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.use_fsync: 0 2024-03-21T21:44:06.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_log_file_size: 0 2024-03-21T21:44:06.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T21:44:06.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T21:44:06.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T21:44:06.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.allow_fallocate: 1 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.use_direct_reads: 0 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.db_log_dir: 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.wal_dir: 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T21:44:06.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.write_buffer_manager: 0x55e4462701e0 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.unordered_write: 0 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T21:44:06.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.row_cache: None 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.wal_filter: None 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.two_write_queues: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.wal_compression: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.atomic_flush: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.log_readahead_size: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T21:44:06.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_background_jobs: 2 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_background_compactions: -1 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_subcompactions: 1 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T21:44:06.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_open_files: -1 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_background_flushes: -1 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Compression algorithms supported: 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: kZSTD supported: 0 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: kXpressCompression supported: 0 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T21:44:06.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: kZlibCompression supported: 1 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: kSnappyCompression supported: 1 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: kLZ4Compression supported: 1 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: kBZip2Compression supported: 0 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.merge_operator: 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_filter: None 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T21:44:06.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e44629fa20) 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x55e4462a3090 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-03-21T21:44:06.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-03-21T21:44:06.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression: NoCompression 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T21:44:06.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.num_levels: 7 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T21:44:06.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T21:44:06.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T21:44:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.table_properties_collectors: 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.inplace_update_support: 0 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.bloom_locality: 0 2024-03-21T21:44:06.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.max_successive_merges: 0 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.ttl: 2592000 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.enable_blob_files: false 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.min_blob_size: 0 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T21:44:06.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 63.sst 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 65, last_sequence is 29280, log_number is 61,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 61 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 505b0e86-ff51-4d09-8599-6a936d9016df 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711057446492588, "job": 1, "event": "recovery_started", "wal_files": [61]} 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #61 mode 2 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711057446494825, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 66, "file_size": 449292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 29281, "largest_seqno": 29771, "table_properties": {"data_size": 446681, "index_size": 1085, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4531, "raw_average_key_size": 22, "raw_value_size": 442569, "raw_average_value_size": 2223, "num_data_blocks": 49, "num_entries": 199, "num_filter_entries": 199, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711057446, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "505b0e86-ff51-4d09-8599-6a936d9016df", "db_session_id": "SQ7X2O3OV03381V1GNT9", "orig_file_number": 66, "seqno_to_time_mapping": "N/A"}} 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711057446494911, "job": 1, "event": "recovery_finished"} 2024-03-21T21:44:06.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/version_set.cc:5047] Creating manifest 68 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000061.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e446392000 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: DB pointer 0x55e44637c000 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T21:44:06.716 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 1/0 438.76 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 306.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: L6 1/0 9.19 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 9.62 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 306.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 306.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T21:44:06.717 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 306.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 97.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 97.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x55e4462a3090#2 capacity: 512.00 MB usage: 1.80 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.20 KB,0.000229478%) Misc(1,0.00 KB,0%) 2024-03-21T21:44:06.718 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:44:06.719 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: mon.a@-1(???) e3 preinit fsid c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: mon.a@-1(???).mds e1 new map 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: mon.a@-1(???).mds e1 print_map 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-03-21T21:44:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: mon.a@-1(???).osd e112 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-21T21:44:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: mon.a@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T21:44:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: mon.a@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T21:44:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: mon.a@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T21:44:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 ceph-mon[85804]: mon.a@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-03-21T21:44:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 bash[85785]: 66c02b062f9ea5faeb7d0c05ca86383330b9fc3ff9559f3a18d86a2a8edd2c7b 2024-03-21T21:44:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:06 smithi028 systemd[1]: Started Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:44:09.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ignoring --setuser ceph since I am not root 2024-03-21T21:44:09.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ignoring --setgroup ceph since I am not root 2024-03-21T21:44:09.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mgr[62478]: -- 172.21.15.28:0/2967155877 <== mon.1 v2:172.21.15.28:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5618674045a0 con 0x5618673d8800 2024-03-21T21:44:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:44:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:44:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:44:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: mon.a calling monitor election 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: fsmap 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: mgrmap e41: y(active, since 3m), standbys: x 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: overall HEALTH_OK 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: from='mgr.25219 ' entity='' 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[85804]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-21T21:44:09.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: mon.a calling monitor election 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: fsmap 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: mgrmap e41: y(active, since 3m), standbys: x 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: overall HEALTH_OK 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: from='mgr.25219 ' entity='' 2024-03-21T21:44:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:09 smithi028 ceph-mon[29568]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-21T21:44:10.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: ignoring --setuser ceph since I am not root 2024-03-21T21:44:10.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: ignoring --setgroup ceph since I am not root 2024-03-21T21:44:10.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mgr[61476]: -- 172.21.15.66:0/373497334 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5575227025a0 con 0x5575226d7000 2024-03-21T21:44:10.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:44:10.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:44:10.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: from='mgr.25219 172.21.15.28:0/1260940633' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:44:10.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:10.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:10.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:10.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: mon.a calling monitor election 2024-03-21T21:44:10.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:44:10.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: monmap e3: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T21:44:10.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: fsmap 2024-03-21T21:44:10.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T21:44:10.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: mgrmap e41: y(active, since 3m), standbys: x 2024-03-21T21:44:10.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: overall HEALTH_OK 2024-03-21T21:44:10.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: from='mgr.25219 ' entity='' 2024-03-21T21:44:10.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:09 smithi066 ceph-mon[68082]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-21T21:44:10.446 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:10.142+0000 7f1630a2b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:44:10.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:10.239+0000 7f1630a2b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:44:10.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:10.139+0000 7f12f975d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T21:44:10.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:10.236+0000 7f12f975d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T21:44:11.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:10.909+0000 7f1630a2b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:44:11.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:11.006+0000 7f1630a2b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:44:11.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:11.151+0000 7f1630a2b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:44:11.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:10.907+0000 7f12f975d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T21:44:11.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:11.005+0000 7f12f975d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T21:44:11.251 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:11.152+0000 7f12f975d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T21:44:11.550 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:11.251+0000 7f1630a2b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:44:11.550 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:11.337+0000 7f1630a2b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:44:11.550 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:11.251+0000 7f12f975d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T21:44:11.550 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:11.338+0000 7f12f975d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T21:44:11.820 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:11 smithi066 ceph-mon[68082]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-21T21:44:11.820 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:11.615+0000 7f12f975d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T21:44:11.820 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:11.728+0000 7f12f975d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:44:11.823 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-mon[29568]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-21T21:44:11.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-mon[85804]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-21T21:44:11.824 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:11.619+0000 7f1630a2b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T21:44:11.824 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:11.733+0000 7f1630a2b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T21:44:12.103 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:11.821+0000 7f1630a2b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:44:12.103 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:11 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:11.920+0000 7f1630a2b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:44:12.103 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:12.101+0000 7f1630a2b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:44:12.104 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:11.819+0000 7f12f975d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T21:44:12.104 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:11.918+0000 7f12f975d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T21:44:12.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:12.103+0000 7f12f975d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T21:44:13.432 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:13.104+0000 7f1630a2b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:44:13.432 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:13.241+0000 7f1630a2b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:44:13.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:13.126+0000 7f12f975d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T21:44:13.500 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:13.263+0000 7f12f975d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T21:44:13.688 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:13.591+0000 7f1630a2b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:44:13.917 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:13.614+0000 7f12f975d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T21:44:13.917 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:13.709+0000 7f12f975d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:44:13.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:13.687+0000 7f1630a2b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T21:44:13.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:13.894+0000 7f1630a2b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:44:14.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:13.915+0000 7f12f975d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:44:14.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:14.111+0000 7f12f975d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:44:14.396 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:14.091+0000 7f1630a2b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T21:44:14.664 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:14.394+0000 7f1630a2b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:44:14.687 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:14.414+0000 7f12f975d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T21:44:14.915 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:14.662+0000 7f1630a2b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:44:14.940 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:14.686+0000 7f12f975d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T21:44:15.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:14.913+0000 7f1630a2b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:44:15.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:14.939+0000 7f12f975d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T21:44:16.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:15 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:15.770+0000 7f1630a2b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:44:16.250 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:15.795+0000 7f12f975d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T21:44:16.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:16.365+0000 7f1630a2b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:44:16.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:16.471+0000 7f1630a2b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:44:16.745 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:16.397+0000 7f12f975d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T21:44:16.745 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:16.502+0000 7f12f975d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T21:44:16.971 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:16.705+0000 7f1630a2b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:44:16.971 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:16] ENGINE Bus STARTING 2024-03-21T21:44:16.971 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CherryPy Checker: 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: The Application mounted at '' has an empty config. 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:16] ENGINE Serving on http://:::9283 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:16] ENGINE Bus STARTED 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: Active manager daemon y restarted 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: Activating manager daemon y 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: mgrmap e44: y(active, starting, since 0.00705644s), standbys: x 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:44:16.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:44:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: Manager daemon y is now available 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: Standby manager daemon x restarted 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: Standby manager daemon x started 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[29568]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: Active manager daemon y restarted 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: Activating manager daemon y 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: mgrmap e44: y(active, starting, since 0.00705644s), standbys: x 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:44:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:44:16.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: Manager daemon y is now available 2024-03-21T21:44:16.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: Standby manager daemon x restarted 2024-03-21T21:44:16.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: Standby manager daemon x started 2024-03-21T21:44:16.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:44:16.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:44:16.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:44:16.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:16 smithi028 ceph-mon[85804]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:44:17.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:16.743+0000 7f12f975d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T21:44:17.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: [21/Mar/2024:21:44:16] ENGINE Bus STARTING 2024-03-21T21:44:17.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: CherryPy Checker: 2024-03-21T21:44:17.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: The Application mounted at '' has an empty config. 2024-03-21T21:44:17.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: 2024-03-21T21:44:17.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: [21/Mar/2024:21:44:16] ENGINE Serving on http://:::9283 2024-03-21T21:44:17.001 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x[61472]: [21/Mar/2024:21:44:16] ENGINE Bus STARTED 2024-03-21T21:44:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: Active manager daemon y restarted 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: Activating manager daemon y 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: mgrmap e44: y(active, starting, since 0.00705644s), standbys: x 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:44:17.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: Manager daemon y is now available 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: Standby manager daemon x restarted 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: Standby manager daemon x started 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T21:44:17.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:16 smithi066 ceph-mon[68082]: from='mgr.? 172.21.15.66:0/3999525099' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T21:44:18.098 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:18.098 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:44:18.098 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:44:18.098 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:44:18.098 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:44:18.098 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:44:18.098 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: [21/Mar/2024:21:44:17] ENGINE Bus STARTING 2024-03-21T21:44:18.098 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: [21/Mar/2024:21:44:17] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:44:18.099 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: [21/Mar/2024:21:44:17] ENGINE Serving on http://172.21.15.28:8765 2024-03-21T21:44:18.099 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: [21/Mar/2024:21:44:17] ENGINE Bus STARTED 2024-03-21T21:44:18.099 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:17 smithi066 ceph-mon[68082]: mgrmap e45: y(active, since 1.01206s), standbys: x 2024-03-21T21:44:18.124 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:17.734+0000 7f15e6bd1700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-21T21:44:18.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:18.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:44:18.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: [21/Mar/2024:21:44:17] ENGINE Bus STARTING 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: [21/Mar/2024:21:44:17] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: [21/Mar/2024:21:44:17] ENGINE Serving on http://172.21.15.28:8765 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: [21/Mar/2024:21:44:17] ENGINE Bus STARTED 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[29568]: mgrmap e45: y(active, since 1.01206s), standbys: x 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:44:18.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:44:18.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T21:44:18.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:44:18.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T21:44:18.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: [21/Mar/2024:21:44:17] ENGINE Bus STARTING 2024-03-21T21:44:18.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: [21/Mar/2024:21:44:17] ENGINE Serving on https://172.21.15.28:7150 2024-03-21T21:44:18.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: [21/Mar/2024:21:44:17] ENGINE Serving on http://172.21.15.28:8765 2024-03-21T21:44:18.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: [21/Mar/2024:21:44:17] ENGINE Bus STARTED 2024-03-21T21:44:18.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:17 smithi028 ceph-mon[85804]: mgrmap e45: y(active, since 1.01206s), standbys: x 2024-03-21T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:18 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:18 smithi028 ceph-mon[85804]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:19.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:18 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:19.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:18 smithi028 ceph-mon[29568]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:18 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:18 smithi066 ceph-mon[68082]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:19.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:19 smithi028 ceph-mon[85804]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:19.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:19 smithi028 ceph-mon[85804]: mgrmap e46: y(active, since 3s), standbys: x 2024-03-21T21:44:19.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:19 smithi028 ceph-mon[29568]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:19.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:19 smithi028 ceph-mon[29568]: mgrmap e46: y(active, since 3s), standbys: x 2024-03-21T21:44:20.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:19 smithi066 ceph-mon[68082]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:20.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:19 smithi066 ceph-mon[68082]: mgrmap e46: y(active, since 3s), standbys: x 2024-03-21T21:44:21.113 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:44:20] "GET /metrics HTTP/1.1" 200 35198 "" "Prometheus/2.43.0" 2024-03-21T21:44:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[29568]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[29568]: mgrmap e47: y(active, since 4s), standbys: x 2024-03-21T21:44:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[85804]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[85804]: mgrmap e47: y(active, since 4s), standbys: x 2024-03-21T21:44:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:21 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:21 smithi066 ceph-mon[68082]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:22.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:21 smithi066 ceph-mon[68082]: mgrmap e47: y(active, since 4s), standbys: x 2024-03-21T21:44:22.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:21 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:21 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:21 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:22.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:21 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:23 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:23 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:23 smithi066 ceph-mon[68082]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:23.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:23 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.744 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:23 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[29568]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:23.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[85804]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:44:23.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:23.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:23 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:25.900 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:25 smithi066 ceph-mon[68082]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:44:26.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:25 smithi028 ceph-mon[85804]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:44:26.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:25 smithi028 ceph-mon[29568]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T21:44:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:44:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:44:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:44:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:44:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[85804]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:44:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[85804]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:44:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:27 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:27 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:27 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:27.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:44:27.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:27 smithi066 ceph-mon[68082]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-21T21:44:27.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:27 smithi066 ceph-mon[68082]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-21T21:44:27.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:27 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:27.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:27 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:44:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:44:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T21:44:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:44:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:44:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:44:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:44:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[29568]: Upgrade: It appears safe to stop mon.c 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:44:28.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:44:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:44:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:44:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T21:44:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-21T21:44:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:28 smithi028 ceph-mon[85804]: Upgrade: It appears safe to stop mon.c 2024-03-21T21:44:28.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: Updating smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: Updating smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:44:28.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T21:44:28.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-21T21:44:28.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:28 smithi066 ceph-mon[68082]: Upgrade: It appears safe to stop mon.c 2024-03-21T21:44:29.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[29568]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:29.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[29568]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:29.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:29.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:29.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[29568]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:29.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:29.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[85804]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:29.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:29.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:29.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:29 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:29.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:29 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:29.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:29 smithi066 ceph-mon[68082]: from='mgr.54108 ' entity='mgr.y' 2024-03-21T21:44:29.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:29 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:29.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:29 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:29.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:29 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:30.484 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:30 smithi028 ceph-mon[29568]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:44:30.484 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:30 smithi028 ceph-mon[29568]: Upgrade: Updating mon.c 2024-03-21T21:44:30.484 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:30 smithi028 ceph-mon[29568]: Deploying daemon mon.c on smithi028 2024-03-21T21:44:30.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:30 smithi028 ceph-mon[85804]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:44:30.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:30 smithi028 ceph-mon[85804]: Upgrade: Updating mon.c 2024-03-21T21:44:30.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:30 smithi028 ceph-mon[85804]: Deploying daemon mon.c on smithi028 2024-03-21T21:44:30.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:30 smithi066 ceph-mon[68082]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T21:44:30.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:30 smithi066 ceph-mon[68082]: Upgrade: Updating mon.c 2024-03-21T21:44:30.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:30 smithi066 ceph-mon[68082]: Deploying daemon mon.c on smithi028 2024-03-21T21:44:31.029 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:44:30] "GET /metrics HTTP/1.1" 200 35198 "" "Prometheus/2.43.0" 2024-03-21T21:44:31.030 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:30 smithi028 systemd[1]: Stopping Ceph mon.c for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:44:31.030 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c[29564]: 2024-03-21T21:44:31.013+0000 7f6121317700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:44:31.030 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c[29564]: 2024-03-21T21:44:31.013+0000 7f6121317700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-21T21:44:31.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:31] ENGINE Bus STOPPING 2024-03-21T21:44:31.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:31] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:44:31.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:31] ENGINE Bus STOPPED 2024-03-21T21:44:31.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:31] ENGINE Bus STARTING 2024-03-21T21:44:31.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:31] ENGINE Serving on http://:::9283 2024-03-21T21:44:31.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:31 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:44:31] ENGINE Bus STARTED 2024-03-21T21:44:31.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:31 smithi028 podman[88021]: 2024-03-21 21:44:31.23780428 +0000 UTC m=+0.325041260 container died 953f34022a7f377a492858bee48273eb689e03b0e644a6bc128972a274d9f87c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-type=git, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, release=754, version=8, io.buildah.version=1.19.8, io.openshift.expose-services=, maintainer=Guillaume Abrioux ) 2024-03-21T21:44:31.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:31 smithi028 podman[88021]: 2024-03-21 21:44:31.604080625 +0000 UTC m=+0.691317586 container cleanup 953f34022a7f377a492858bee48273eb689e03b0e644a6bc128972a274d9f87c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, ceph=True, name=centos-stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0) 2024-03-21T21:44:31.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:31 smithi028 bash[88021]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c 2024-03-21T21:44:32.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:31 smithi028 podman[88046]: 2024-03-21 21:44:31.980089215 +0000 UTC m=+0.742754284 container remove 953f34022a7f377a492858bee48273eb689e03b0e644a6bc128972a274d9f87c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_CLEAN=True, release=754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, distribution-scope=public) 2024-03-21T21:44:32.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-21T21:44:32.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 systemd[1]: Stopped Ceph mon.c for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:44:32.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.c.service: Consumed 16.675s CPU time. 2024-03-21T21:44:32.326 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:44:32.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 systemd[1]: Starting Ceph mon.c for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:44:32.870 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 podman[88222]: 2024-03-21 21:44:32.598285873 +0000 UTC m=+0.102954651 container create d1de564deab3d7ce463fc18342ed228fe9a002e768012caa3816433980466a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T21:44:32.870 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 podman[88222]: 2024-03-21 21:44:32.514546007 +0000 UTC m=+0.019214785 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:44:32.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 podman[88222]: 2024-03-21 21:44:32.842785805 +0000 UTC m=+0.347454595 container init d1de564deab3d7ce463fc18342ed228fe9a002e768012caa3816433980466a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-21T21:44:32.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 podman[88222]: 2024-03-21 21:44:32.845970632 +0000 UTC m=+0.350639410 container start d1de564deab3d7ce463fc18342ed228fe9a002e768012caa3816433980466a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:44:33.028 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:44:33.028 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (2m) 11s ago 24m 28.8M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:44:33.028 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (2m) 11s ago 24m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:44:33.028 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (118s) 11s ago 23m 48.9M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (3m) 11s ago 29m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (13m) 11s ago 31m 475M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (26s) 11s ago 31m 31.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (78s) 11s ago 29m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 starting - - - 2048M 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (3m) 11s ago 25m 14.8M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (3m) 11s ago 25m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (29m) 11s ago 29m 54.2M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (28m) 11s ago 28m 55.2M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (28m) 11s ago 28m 53.3M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (27m) 11s ago 27m 53.3M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (27m) 11s ago 27m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:44:33.029 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (27m) 11s ago 27m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:44:33.030 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (26m) 11s ago 26m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:44:33.030 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (26m) 11s ago 26m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:44:33.030 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (2m) 11s ago 25m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:44:33.030 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (23m) 11s ago 23m 77.6M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:44:33.030 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (23m) 11s ago 23m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:44:33.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T21:44:33.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev), process ceph-mon, pid 2 2024-03-21T21:44:33.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: pidfile_write: ignore empty --pid-file 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: load: jerasure load: lrc 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: RocksDB version: 7.9.2 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Git sha 0 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Compile date 2024-03-21 16:40:47 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: DB SUMMARY 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: DB Session ID: G9BMR9DJU83J5TH9XOWB 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: CURRENT file: CURRENT 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3195 Bytes 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000060.sst 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000058.log size: 2319688 ; 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.error_if_exists: 0 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.create_if_missing: 0 2024-03-21T21:44:33.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.paranoid_checks: 1 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.env: 0x55e1d5785b20 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.info_log: 0x55e1d7923920 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.statistics: (nil) 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.use_fsync: 0 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_log_file_size: 0 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T21:44:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.allow_fallocate: 1 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.use_direct_reads: 0 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.db_log_dir: 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.wal_dir: 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T21:44:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.write_buffer_manager: 0x55e1d78f41e0 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.unordered_write: 0 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T21:44:33.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.row_cache: None 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.wal_filter: None 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.two_write_queues: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.wal_compression: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.atomic_flush: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.log_readahead_size: 0 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T21:44:33.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T21:44:33.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T21:44:33.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T21:44:33.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T21:44:33.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T21:44:33.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T21:44:33.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_background_jobs: 2 2024-03-21T21:44:33.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_background_compactions: -1 2024-03-21T21:44:33.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_subcompactions: 1 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_open_files: -1 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_background_flushes: -1 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Compression algorithms supported: 2024-03-21T21:44:33.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: kZSTD supported: 0 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: kXpressCompression supported: 0 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: kZlibCompression supported: 1 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: kSnappyCompression supported: 1 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: kLZ4Compression supported: 1 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: kBZip2Compression supported: 0 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.merge_operator: 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_filter: None 2024-03-21T21:44:33.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e1d7923a20) 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: index_type: 0 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: data_block_index_type: 0 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: index_shortening: 1 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: checksum: 4 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: no_block_cache: 0 2024-03-21T21:44:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache: 0x55e1d7927090 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache_options: 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: capacity : 536870912 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: num_shard_bits : 4 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: strict_capacity_limit : 0 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache_compressed: (nil) 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: persistent_cache: (nil) 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_size: 4096 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_size_deviation: 10 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_restart_interval: 16 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: index_block_restart_interval: 1 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: metadata_block_size: 4096 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: partition_filters: 0 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: use_delta_encoding: 1 2024-03-21T21:44:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout: filter_policy: bloomfilter 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: whole_key_filtering: 1 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: verify_compression: 0 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: format_version: 5 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: enable_index_compression: 1 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_align: 0 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: max_auto_readahead_size: 262144 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: prepopulate_block_cache: 0 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: initial_auto_readahead_size: 8192 2024-03-21T21:44:33.209 INFO:journalctl@ceph.mon.c.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression: NoCompression 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.num_levels: 7 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T21:44:33.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T21:44:33.211 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T21:44:33.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T21:44:33.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.table_properties_collectors: 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.inplace_update_support: 0 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T21:44:33.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.bloom_locality: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.max_successive_merges: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.ttl: 2592000 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.enable_blob_files: false 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.min_blob_size: 0 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T21:44:33.215 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 62, last_sequence is 32702, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d60ddcc2-5987-4690-940a-b495dff9009b 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711057472876460, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711057472884533, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 1478089, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 32707, "largest_seqno": 33108, "table_properties": {"data_size": 1475237, "index_size": 1326, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4666, "raw_average_key_size": 24, "raw_value_size": 1470859, "raw_average_value_size": 7621, "num_data_blocks": 59, "num_entries": 193, "num_filter_entries": 193, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711057472, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d60ddcc2-5987-4690-940a-b495dff9009b", "db_session_id": "G9BMR9DJU83J5TH9XOWB", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-03-21T21:44:33.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711057472884641, "job": 1, "event": "recovery_finished"} 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e1d7a16000 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: DB pointer 0x55e1d7a00000 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** DB Stats ** 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T21:44:33.217 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: L0 1/0 1.41 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 432.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: L6 1/0 10.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: Sum 2/0 11.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 432.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 432.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T21:44:33.218 INFO:journalctl@ceph.mon.c.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 432.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative compaction: 0.00 GB write, 132.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval compaction: 0.00 GB write, 132.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T21:44:33.219 INFO:journalctl@ceph.mon.c.smithi028.stdout: Block cache BinnedLRUCache@0x55e1d7927090#2 capacity: 512.00 MB usage: 2.05 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.45 KB,0.000277162%) Misc(1,0.00 KB,0%) 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: starting mon.c rank 1 at public addrs [v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0] at bind addrs [v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: mon.c@-1(???) e3 preinit fsid c10cc158-e7c7-11ee-95cd-87774f69a715 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: mon.c@-1(???).mds e1 new map 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: mon.c@-1(???).mds e1 print_map 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout: e1 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout: legacy client fscid: -1 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout: No filesystems configured 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: mon.c@-1(???).osd e113 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-21T21:44:33.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: mon.c@-1(???).osd e113 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T21:44:33.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: mon.c@-1(???).osd e113 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T21:44:33.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: mon.c@-1(???).osd e113 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T21:44:33.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 ceph-mon[88264]: mon.c@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-03-21T21:44:33.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 bash[88222]: d1de564deab3d7ce463fc18342ed228fe9a002e768012caa3816433980466a07 2024-03-21T21:44:33.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:32 smithi028 systemd[1]: Started Ceph mon.c for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:44:33.434 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:44:33.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:44:33.436 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:44:33.436 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-21T21:44:33.436 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 4 2024-03-21T21:44:33.436 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:44:33.436 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:44:33.778 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:44:33.778 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:44:33.778 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:44:33.778 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi028", 2024-03-21T21:44:33.778 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:44:33.778 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "1/2 daemons upgraded", 2024-03-21T21:44:33.778 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading mon daemons", 2024-03-21T21:44:33.778 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:44:33.778 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:44:38.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:37 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:44:37.892+0000 7f15e6bd1700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-21T21:44:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: from='client.54159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:44:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: mon.a calling monitor election 2024-03-21T21:44:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: mon.b calling monitor election 2024-03-21T21:44:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: mon.c calling monitor election 2024-03-21T21:44:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: mon.a is new leader, mons a,b in quorum (ranks 0,2) 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: overall HEALTH_OK 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: mon.a calling monitor election 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: monmap e4: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: fsmap 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: mgrmap e47: y(active, since 22s), standbys: x 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: overall HEALTH_OK 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: from='client.54159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: mon.a calling monitor election 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: mon.b calling monitor election 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: mon.c calling monitor election 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:39.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: mon.a is new leader, mons a,b in quorum (ranks 0,2) 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: overall HEALTH_OK 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: mon.a calling monitor election 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: monmap e4: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: fsmap 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: mgrmap e47: y(active, since 22s), standbys: x 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: overall HEALTH_OK 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:39.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: from='client.54159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: mon.a calling monitor election 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: mon.b calling monitor election 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: mon.c calling monitor election 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: mon.a is new leader, mons a,b in quorum (ranks 0,2) 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: overall HEALTH_OK 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: mon.a calling monitor election 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: monmap e4: 3 mons at {a=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],b=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0],c=[v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: fsmap 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: mgrmap e47: y(active, since 22s), standbys: x 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: overall HEALTH_OK 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:40.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:44:40.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:40 smithi028 ceph-mon[88264]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:40.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:40 smithi028 ceph-mon[85804]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:41.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:40 smithi066 ceph-mon[68082]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:41.098 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:44:40] "GET /metrics HTTP/1.1" 200 38005 "" "Prometheus/2.43.0" 2024-03-21T21:44:42.781 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:42 smithi028 ceph-mon[88264]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:42.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:42 smithi028 ceph-mon[85804]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:43.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:42 smithi066 ceph-mon[68082]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:44.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:44 smithi028 ceph-mon[85804]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:44.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:44 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:44.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:44 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:44.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:44 smithi028 ceph-mon[88264]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:44.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:44 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:44.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:44 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:45.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:44 smithi066 ceph-mon[68082]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:45.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:44 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:45.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:44 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:46.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:46 smithi028 ceph-mon[85804]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:46.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:46.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:46.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:46 smithi028 ceph-mon[88264]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:46.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:46.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:46.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:46 smithi066 ceph-mon[68082]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:46.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:46.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:47.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:44:47.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:44:47.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:44:48.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:48 smithi028 ceph-mon[88264]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:48.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:48 smithi028 ceph-mon[85804]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:48.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:48 smithi066 ceph-mon[68082]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:49.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:49.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:49.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: Detected new or changed devices on smithi028 2024-03-21T21:44:49.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:49.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:49.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:49.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:44:49.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: Detected new or changed devices on smithi028 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:50.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:50.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: Detected new or changed devices on smithi028 2024-03-21T21:44:50.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:50.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:50.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:50.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:44:50.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:50.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:50.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:50.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:50.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[88264]: Reconfiguring mon.a (monmap changed)... 2024-03-21T21:44:50.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[88264]: Reconfiguring daemon mon.a on smithi028 2024-03-21T21:44:50.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[88264]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:50.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:50.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:50.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:44:50.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:44:50.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:50.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[85804]: Reconfiguring mon.a (monmap changed)... 2024-03-21T21:44:50.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[85804]: Reconfiguring daemon mon.a on smithi028 2024-03-21T21:44:50.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[85804]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:50.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:50.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:50.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:44:50.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:44:50.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:50 smithi066 ceph-mon[68082]: Reconfiguring mon.a (monmap changed)... 2024-03-21T21:44:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:50 smithi066 ceph-mon[68082]: Reconfiguring daemon mon.a on smithi028 2024-03-21T21:44:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:50 smithi066 ceph-mon[68082]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:44:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:44:51.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:51.190 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:44:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:44:50] "GET /metrics HTTP/1.1" 200 38013 "" "Prometheus/2.43.0" 2024-03-21T21:44:51.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:51 smithi028 ceph-mon[85804]: Reconfiguring mgr.y (monmap changed)... 2024-03-21T21:44:51.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:51 smithi028 ceph-mon[85804]: Reconfiguring daemon mgr.y on smithi028 2024-03-21T21:44:51.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:51 smithi028 ceph-mon[88264]: Reconfiguring mgr.y (monmap changed)... 2024-03-21T21:44:51.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:51 smithi028 ceph-mon[88264]: Reconfiguring daemon mgr.y on smithi028 2024-03-21T21:44:52.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:51 smithi066 ceph-mon[68082]: Reconfiguring mgr.y (monmap changed)... 2024-03-21T21:44:52.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:51 smithi066 ceph-mon[68082]: Reconfiguring daemon mgr.y on smithi028 2024-03-21T21:44:52.791 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[88264]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:52.791 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:52.791 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:52.791 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:52.791 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:52.791 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:52.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[85804]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:52.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:52.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:52.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:52.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:52.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:53.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:52 smithi066 ceph-mon[68082]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:44:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:44:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:53.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:53 smithi028 ceph-mon[88264]: Reconfiguring mon.c (monmap changed)... 2024-03-21T21:44:53.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:53 smithi028 ceph-mon[88264]: Reconfiguring daemon mon.c on smithi028 2024-03-21T21:44:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:53 smithi028 ceph-mon[85804]: Reconfiguring mon.c (monmap changed)... 2024-03-21T21:44:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:53 smithi028 ceph-mon[85804]: Reconfiguring daemon mon.c on smithi028 2024-03-21T21:44:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:53 smithi066 ceph-mon[68082]: Reconfiguring mon.c (monmap changed)... 2024-03-21T21:44:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:53 smithi066 ceph-mon[68082]: Reconfiguring daemon mon.c on smithi028 2024-03-21T21:44:54.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[85804]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:54.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:54.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:54.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T21:44:54.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[88264]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:54.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:54.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:54.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T21:44:54.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:54 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:54 smithi066 ceph-mon[68082]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:54 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:54 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:54 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T21:44:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:54 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:55.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[85804]: Reconfiguring osd.0 (monmap changed)... 2024-03-21T21:44:55.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[85804]: Reconfiguring daemon osd.0 on smithi028 2024-03-21T21:44:55.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:55.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[88264]: Reconfiguring osd.0 (monmap changed)... 2024-03-21T21:44:55.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[88264]: Reconfiguring daemon osd.0 on smithi028 2024-03-21T21:44:55.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:55.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:55.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T21:44:55.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:55.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:55.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T21:44:55.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:55 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:56.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:55 smithi066 ceph-mon[68082]: Reconfiguring osd.0 (monmap changed)... 2024-03-21T21:44:56.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:55 smithi066 ceph-mon[68082]: Reconfiguring daemon osd.0 on smithi028 2024-03-21T21:44:56.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:55 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:56.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:55 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:56.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:55 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T21:44:56.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:55 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:56.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:56 smithi028 ceph-mon[85804]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:56.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:56 smithi028 ceph-mon[85804]: Reconfiguring osd.1 (monmap changed)... 2024-03-21T21:44:56.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:56 smithi028 ceph-mon[85804]: Reconfiguring daemon osd.1 on smithi028 2024-03-21T21:44:56.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:56 smithi028 ceph-mon[88264]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:56.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:56 smithi028 ceph-mon[88264]: Reconfiguring osd.1 (monmap changed)... 2024-03-21T21:44:56.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:56 smithi028 ceph-mon[88264]: Reconfiguring daemon osd.1 on smithi028 2024-03-21T21:44:57.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:56 smithi066 ceph-mon[68082]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:44:57.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:56 smithi066 ceph-mon[68082]: Reconfiguring osd.1 (monmap changed)... 2024-03-21T21:44:57.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:56 smithi066 ceph-mon[68082]: Reconfiguring daemon osd.1 on smithi028 2024-03-21T21:44:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[88264]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[88264]: Reconfiguring osd.2 (monmap changed)... 2024-03-21T21:44:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T21:44:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[88264]: Reconfiguring daemon osd.2 on smithi028 2024-03-21T21:44:58.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[85804]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:58.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:58.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:58.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[85804]: Reconfiguring osd.2 (monmap changed)... 2024-03-21T21:44:58.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T21:44:58.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:58.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:58 smithi028 ceph-mon[85804]: Reconfiguring daemon osd.2 on smithi028 2024-03-21T21:44:58.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:58 smithi066 ceph-mon[68082]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:44:58.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:58.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:58.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:58 smithi066 ceph-mon[68082]: Reconfiguring osd.2 (monmap changed)... 2024-03-21T21:44:58.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T21:44:58.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:58.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:58 smithi066 ceph-mon[68082]: Reconfiguring daemon osd.2 on smithi028 2024-03-21T21:44:59.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:59.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:59.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:59.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T21:44:59.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:59.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:59.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:59.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:59.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T21:44:59.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:44:59 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:44:59.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:59 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:44:59.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:59 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:59.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:59 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:44:59.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:59 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T21:44:59.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:44:59 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:00.260 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:00 smithi028 ceph-mon[85804]: Reconfiguring osd.3 (monmap changed)... 2024-03-21T21:45:00.260 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:00 smithi028 ceph-mon[85804]: Reconfiguring daemon osd.3 on smithi028 2024-03-21T21:45:00.260 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:00 smithi028 ceph-mon[85804]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:00.260 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:00 smithi028 ceph-mon[88264]: Reconfiguring osd.3 (monmap changed)... 2024-03-21T21:45:00.260 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:00 smithi028 ceph-mon[88264]: Reconfiguring daemon osd.3 on smithi028 2024-03-21T21:45:00.260 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:00 smithi028 ceph-mon[88264]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:00.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:00 smithi066 ceph-mon[68082]: Reconfiguring osd.3 (monmap changed)... 2024-03-21T21:45:00.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:00 smithi066 ceph-mon[68082]: Reconfiguring daemon osd.3 on smithi028 2024-03-21T21:45:00.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:00 smithi066 ceph-mon[68082]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:01.082 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:45:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:45:00] "GET /metrics HTTP/1.1" 200 38013 "" "Prometheus/2.43.0" 2024-03-21T21:45:01.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:01.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:01.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[85804]: Reconfiguring rgw.foo.smithi028.qubqiy (monmap changed)... 2024-03-21T21:45:01.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:45:01.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:01.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[85804]: Reconfiguring daemon rgw.foo.smithi028.qubqiy on smithi028 2024-03-21T21:45:01.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:01.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:01.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[88264]: Reconfiguring rgw.foo.smithi028.qubqiy (monmap changed)... 2024-03-21T21:45:01.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:45:01.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:01.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:01 smithi028 ceph-mon[88264]: Reconfiguring daemon rgw.foo.smithi028.qubqiy on smithi028 2024-03-21T21:45:01.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:01.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:01.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:01 smithi066 ceph-mon[68082]: Reconfiguring rgw.foo.smithi028.qubqiy (monmap changed)... 2024-03-21T21:45:01.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:45:01.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:01.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:01 smithi066 ceph-mon[68082]: Reconfiguring daemon rgw.foo.smithi028.qubqiy on smithi028 2024-03-21T21:45:02.655 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:02 smithi066 ceph-mon[68082]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:02.655 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:02.655 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:02.655 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:02.655 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:45:02.655 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:45:02.656 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[85804]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:45:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:45:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[88264]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T21:45:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T21:45:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:03.662 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:03 smithi066 ceph-mon[68082]: Reconfiguring mon.b (monmap changed)... 2024-03-21T21:45:03.663 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:03 smithi066 ceph-mon[68082]: Reconfiguring daemon mon.b on smithi066 2024-03-21T21:45:03.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:03 smithi028 ceph-mon[85804]: Reconfiguring mon.b (monmap changed)... 2024-03-21T21:45:03.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:03 smithi028 ceph-mon[85804]: Reconfiguring daemon mon.b on smithi066 2024-03-21T21:45:03.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:03 smithi028 ceph-mon[88264]: Reconfiguring mon.b (monmap changed)... 2024-03-21T21:45:03.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:03 smithi028 ceph-mon[88264]: Reconfiguring daemon mon.b on smithi066 2024-03-21T21:45:04.140 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:45:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[88264]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:45:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:45:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[85804]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:04.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:04.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:45:04.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:45:04.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:04 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:04.615 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:04 smithi066 ceph-mon[68082]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:04.615 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:04 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:04.615 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:04 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:04.615 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:04 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T21:45:04.615 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:04 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T21:45:04.615 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:04 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:04.832 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:45:04.832 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (3m) 20s ago 25m 28.9M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:45:04.832 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (2m) 43s ago 24m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:45:04.832 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (2m) 20s ago 24m 53.9M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:45:04.833 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (4m) 43s ago 29m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:45:04.833 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (14m) 20s ago 31m 496M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:45:04.833 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (58s) 20s ago 31m 40.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:45:04.833 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (110s) 43s ago 30m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:45:04.833 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (32s) 20s ago 30m 20.1M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:45:04.833 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (3m) 20s ago 25m 15.4M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:45:04.833 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (3m) 43s ago 25m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:45:04.833 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (29m) 20s ago 29m 54.4M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:45:04.834 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (29m) 20s ago 29m 55.6M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:45:04.834 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (28m) 20s ago 28m 53.5M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:45:04.834 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (28m) 20s ago 28m 53.6M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:45:04.834 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (27m) 43s ago 27m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:45:04.834 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (27m) 43s ago 27m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:45:04.834 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (27m) 43s ago 27m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:45:04.834 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (26m) 43s ago 26m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:45:04.834 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (3m) 43s ago 26m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:45:04.835 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (24m) 20s ago 24m 78.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:45:04.835 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (24m) 43s ago 24m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:45:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:45:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:45:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:45:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-21T21:45:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 5 2024-03-21T21:45:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:45:05.241 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:45:05.585 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi028", 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading mon daemons", 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:45:05.586 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:45:05.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[88264]: Reconfiguring mgr.x (monmap changed)... 2024-03-21T21:45:05.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[88264]: Reconfiguring daemon mgr.x on smithi066 2024-03-21T21:45:05.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[88264]: from='client.54165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:05.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[88264]: from='client.54171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:05.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/3056575644' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:05.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[85804]: Reconfiguring mgr.x (monmap changed)... 2024-03-21T21:45:05.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[85804]: Reconfiguring daemon mgr.x on smithi066 2024-03-21T21:45:05.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[85804]: from='client.54165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:05.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[85804]: from='client.54171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:05.588 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:05 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/3056575644' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:05.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:05 smithi066 ceph-mon[68082]: Reconfiguring mgr.x (monmap changed)... 2024-03-21T21:45:05.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:05 smithi066 ceph-mon[68082]: Reconfiguring daemon mgr.x on smithi066 2024-03-21T21:45:05.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:05 smithi066 ceph-mon[68082]: from='client.54165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:05.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:05 smithi066 ceph-mon[68082]: from='client.54171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:05.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:05 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/3056575644' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:06.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:06 smithi066 ceph-mon[68082]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:06.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:06 smithi066 ceph-mon[68082]: from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:06.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:06.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:06.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:06 smithi066 ceph-mon[68082]: Reconfiguring osd.4 (monmap changed)... 2024-03-21T21:45:06.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T21:45:06.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:06.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:06 smithi066 ceph-mon[68082]: Reconfiguring daemon osd.4 on smithi066 2024-03-21T21:45:06.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:06 smithi066 ceph-mon[68082]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[85804]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[85804]: from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[85804]: Reconfiguring osd.4 (monmap changed)... 2024-03-21T21:45:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T21:45:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[85804]: Reconfiguring daemon osd.4 on smithi066 2024-03-21T21:45:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[85804]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[88264]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[88264]: from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[88264]: Reconfiguring osd.4 (monmap changed)... 2024-03-21T21:45:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T21:45:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[88264]: Reconfiguring daemon osd.4 on smithi066 2024-03-21T21:45:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:06 smithi028 ceph-mon[88264]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:08.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:08 smithi066 ceph-mon[68082]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:08.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:08 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:08.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:08 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:08.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:08 smithi066 ceph-mon[68082]: Reconfiguring osd.5 (monmap changed)... 2024-03-21T21:45:08.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:08 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T21:45:08.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:08 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:08.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:08 smithi066 ceph-mon[68082]: Reconfiguring daemon osd.5 on smithi066 2024-03-21T21:45:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[85804]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[85804]: Reconfiguring osd.5 (monmap changed)... 2024-03-21T21:45:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T21:45:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:08.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[85804]: Reconfiguring daemon osd.5 on smithi066 2024-03-21T21:45:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[88264]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[88264]: Reconfiguring osd.5 (monmap changed)... 2024-03-21T21:45:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T21:45:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:08 smithi028 ceph-mon[88264]: Reconfiguring daemon osd.5 on smithi066 2024-03-21T21:45:09.625 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:09 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:09.625 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:09 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:09.625 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:09 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:09.625 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:09 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T21:45:09.625 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:09 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:09.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:09.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:09.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:09.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T21:45:09.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:09.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:09.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:09.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:09.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T21:45:09.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:09 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:10.593 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:10 smithi066 ceph-mon[68082]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:10.593 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:10 smithi066 ceph-mon[68082]: Reconfiguring osd.6 (monmap changed)... 2024-03-21T21:45:10.593 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:10 smithi066 ceph-mon[68082]: Reconfiguring daemon osd.6 on smithi066 2024-03-21T21:45:10.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:10 smithi028 ceph-mon[85804]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:10.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:10 smithi028 ceph-mon[85804]: Reconfiguring osd.6 (monmap changed)... 2024-03-21T21:45:10.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:10 smithi028 ceph-mon[85804]: Reconfiguring daemon osd.6 on smithi066 2024-03-21T21:45:10.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:10 smithi028 ceph-mon[88264]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:10.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:10 smithi028 ceph-mon[88264]: Reconfiguring osd.6 (monmap changed)... 2024-03-21T21:45:10.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:10 smithi028 ceph-mon[88264]: Reconfiguring daemon osd.6 on smithi066 2024-03-21T21:45:11.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:45:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:45:10] "GET /metrics HTTP/1.1" 200 38014 "" "Prometheus/2.43.0" 2024-03-21T21:45:12.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:11 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:12.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:11 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:12.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:11 smithi066 ceph-mon[68082]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:12.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:11 smithi066 ceph-mon[68082]: Reconfiguring osd.7 (monmap changed)... 2024-03-21T21:45:12.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:11 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T21:45:12.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:11 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:12.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:11 smithi066 ceph-mon[68082]: Reconfiguring daemon osd.7 on smithi066 2024-03-21T21:45:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[85804]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[85804]: Reconfiguring osd.7 (monmap changed)... 2024-03-21T21:45:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T21:45:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[85804]: Reconfiguring daemon osd.7 on smithi066 2024-03-21T21:45:12.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:12.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:12.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[88264]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:12.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[88264]: Reconfiguring osd.7 (monmap changed)... 2024-03-21T21:45:12.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T21:45:12.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:12.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:11 smithi028 ceph-mon[88264]: Reconfiguring daemon osd.7 on smithi066 2024-03-21T21:45:13.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:13.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:13.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:13 smithi066 ceph-mon[68082]: Reconfiguring rgw.foo.smithi066.gkaqro (monmap changed)... 2024-03-21T21:45:13.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:45:13.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:13.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:13 smithi066 ceph-mon[68082]: Reconfiguring daemon rgw.foo.smithi066.gkaqro on smithi066 2024-03-21T21:45:13.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:13.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:13.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[88264]: Reconfiguring rgw.foo.smithi066.gkaqro (monmap changed)... 2024-03-21T21:45:13.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:45:13.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:13.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[88264]: Reconfiguring daemon rgw.foo.smithi066.gkaqro on smithi066 2024-03-21T21:45:13.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:13.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:13.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[85804]: Reconfiguring rgw.foo.smithi066.gkaqro (monmap changed)... 2024-03-21T21:45:13.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:45:13.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:13.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:13 smithi028 ceph-mon[85804]: Reconfiguring daemon rgw.foo.smithi066.gkaqro on smithi066 2024-03-21T21:45:14.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:14.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:45:14.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:45:14.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:45:14.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:14 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:15.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:45:15.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:45:15.005 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:15.006 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:15.007 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:45:15.007 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.007 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:15.007 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:45:15.007 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:45:15.007 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:14 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mon 2024-03-21T21:45:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all crash 2024-03-21T21:45:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mds 2024-03-21T21:45:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:45:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:45:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all nfs 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all node-exporter 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all prometheus 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all alertmanager 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all grafana 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all loki 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all promtail 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Finalizing container_image settings 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[88264]: Upgrade: Complete! 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mon 2024-03-21T21:45:15.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all crash 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mds 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all nfs 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all node-exporter 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all prometheus 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all alertmanager 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all grafana 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all loki 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all promtail 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Finalizing container_image settings 2024-03-21T21:45:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:15 smithi028 ceph-mon[85804]: Upgrade: Complete! 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mon 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all crash 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mds 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nfs 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all node-exporter 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all prometheus 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all alertmanager 2024-03-21T21:45:16.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all grafana 2024-03-21T21:45:16.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all loki 2024-03-21T21:45:16.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all promtail 2024-03-21T21:45:16.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Finalizing container_image settings 2024-03-21T21:45:16.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:15 smithi066 ceph-mon[68082]: Upgrade: Complete! 2024-03-21T21:45:16.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:16 smithi028 ceph-mon[88264]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:16 smithi028 ceph-mon[85804]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:16 smithi066 ceph-mon[68082]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:18.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:18.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:18 smithi028 ceph-mon[85804]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:18.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:18 smithi028 ceph-mon[88264]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:19.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:18 smithi066 ceph-mon[68082]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:19.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:19 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:19.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:19 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:19.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:19 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:20.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:20 smithi028 ceph-mon[88264]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:20.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:20 smithi028 ceph-mon[85804]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:21.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:20 smithi066 ceph-mon[68082]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:21.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:45:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:45:20] "GET /metrics HTTP/1.1" 200 38012 "" "Prometheus/2.43.0" 2024-03-21T21:45:22.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:22 smithi028 ceph-mon[88264]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:22 smithi028 ceph-mon[85804]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:23.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:22 smithi066 ceph-mon[68082]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:24.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:24 smithi028 ceph-mon[88264]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:24.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:24 smithi028 ceph-mon[85804]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:25.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:24 smithi066 ceph-mon[68082]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:26.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:26 smithi028 ceph-mon[88264]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:26.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:26 smithi028 ceph-mon[85804]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:27.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:26 smithi066 ceph-mon[68082]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:28.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:28 smithi028 ceph-mon[85804]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:28.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:28 smithi028 ceph-mon[88264]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:29.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:28 smithi066 ceph-mon[68082]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:29.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:29 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:29 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:29 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:30.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:30 smithi028 ceph-mon[85804]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:30.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:45:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:45:30] "GET /metrics HTTP/1.1" 200 38012 "" "Prometheus/2.43.0" 2024-03-21T21:45:30.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:30 smithi028 ceph-mon[88264]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:30 smithi066 ceph-mon[68082]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:32.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:32 smithi028 ceph-mon[85804]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:32.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:32.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:32 smithi028 ceph-mon[88264]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:32.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:32 smithi066 ceph-mon[68082]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:33.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:34.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:34 smithi028 ceph-mon[85804]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:34.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:34 smithi028 ceph-mon[88264]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:35.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:34 smithi066 ceph-mon[68082]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:36.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:45:36.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:36 smithi028 ceph-mon[85804]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:36.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:36 smithi028 ceph-mon[88264]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:37.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:36 smithi066 ceph-mon[68082]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:37.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:37 smithi028 ceph-mon[88264]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:37.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:37 smithi028 ceph-mon[85804]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:37 smithi066 ceph-mon[68082]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:38.802 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:45:38.802 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (3m) 54s ago 25m 28.9M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:45:38.802 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (3m) 77s ago 25m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:45:38.802 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (3m) 54s ago 24m 53.9M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (4m) 77s ago 30m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (14m) 54s ago 32m 496M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (92s) 54s ago 32m 40.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (2m) 77s ago 30m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (66s) 54s ago 30m 20.1M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (4m) 54s ago 26m 15.4M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (4m) 77s ago 26m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (30m) 54s ago 30m 54.4M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (29m) 54s ago 29m 55.6M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (29m) 54s ago 29m 53.5M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (28m) 54s ago 28m 53.6M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (28m) 77s ago 28m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (28m) 77s ago 28m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (27m) 77s ago 27m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:45:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (27m) 77s ago 27m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:45:38.804 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (3m) 77s ago 26m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:45:38.804 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (25m) 54s ago 25m 78.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:45:38.804 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (24m) 77s ago 24m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:45:38.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:38 smithi028 ceph-mon[88264]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:38.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:38 smithi028 ceph-mon[85804]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:39.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:38 smithi066 ceph-mon[68082]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:39.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-03-21T21:45:39.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:39 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:39 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:39 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:40.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:40 smithi028 ceph-mon[88264]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:40.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:40 smithi028 ceph-mon[88264]: from='client.54201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:40.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:40 smithi028 ceph-mon[85804]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:40.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:40 smithi028 ceph-mon[85804]: from='client.54201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:41.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:40 smithi066 ceph-mon[68082]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:40 smithi066 ceph-mon[68082]: from='client.54201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:41.118 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:45:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:45:40] "GET /metrics HTTP/1.1" 200 38007 "" "Prometheus/2.43.0" 2024-03-21T21:45:41.465 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:45:41.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:41 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/891880897' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:41.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:41 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/891880897' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:41 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/891880897' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:42.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-03-21T21:45:42.870 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:42 smithi028 ceph-mon[88264]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:42.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:42 smithi028 ceph-mon[85804]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:43.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:42 smithi066 ceph-mon[68082]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:44.043 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)" 2024-03-21T21:45:44.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-03-21T21:45:44.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:44 smithi028 ceph-mon[88264]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:44.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:44 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/4033201888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:44.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:44 smithi028 ceph-mon[85804]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:44.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:44 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/4033201888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:45.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:44 smithi066 ceph-mon[68082]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:45.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:44 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/4033201888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:45:46.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:46 smithi028 ceph-mon[88264]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:45:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:46 smithi028 ceph-mon[85804]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:45:47.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:46 smithi066 ceph-mon[68082]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:45:47.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:47.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:48.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:45:48.740 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:48 smithi028 ceph-mon[88264]: from='client.54219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:48.740 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:48 smithi028 ceph-mon[88264]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:45:48.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:48 smithi028 ceph-mon[85804]: from='client.54219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:48.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:48 smithi028 ceph-mon[85804]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:45:48.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:48 smithi066 ceph-mon[68082]: from='client.54219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:48.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:48 smithi066 ceph-mon[68082]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:45:48.798 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:45:49.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade status' 2024-03-21T21:45:49.919 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:49 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:49.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:49 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:50.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:49 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:50.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:50 smithi028 ceph-mon[85804]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:50.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:50 smithi028 ceph-mon[88264]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:51.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:50 smithi066 ceph-mon[68082]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:51.104 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:45:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:45:50] "GET /metrics HTTP/1.1" 200 37941 "" "Prometheus/2.43.0" 2024-03-21T21:45:51.444 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:45:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": null, 2024-03-21T21:45:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": false, 2024-03-21T21:45:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "", 2024-03-21T21:45:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:45:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": null, 2024-03-21T21:45:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-03-21T21:45:51.445 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:45:51.445 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:45:52.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:45:52.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:52 smithi028 ceph-mon[88264]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:45:52.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:52 smithi028 ceph-mon[88264]: from='client.54225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:52.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:52 smithi028 ceph-mon[85804]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:45:52.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:52 smithi028 ceph-mon[85804]: from='client.54225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:53.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:52 smithi066 ceph-mon[68082]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:45:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:52 smithi066 ceph-mon[68082]: from='client.54225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:54.042 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-03-21T21:45:54.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:54 smithi028 ceph-mon[85804]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:54.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:54 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/621823620' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:45:54.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:54 smithi028 ceph-mon[88264]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:54.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:54 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/621823620' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:45:54.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-03-21T21:45:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:54 smithi066 ceph-mon[68082]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:45:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:54 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/621823620' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:45:56.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:56 smithi028 ceph-mon[88264]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:56.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:56 smithi028 ceph-mon[85804]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:57.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:56 smithi066 ceph-mon[68082]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:58.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:58 smithi028 ceph-mon[88264]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:58.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:58 smithi028 ceph-mon[88264]: from='client.54237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:58.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:58 smithi028 ceph-mon[85804]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:58.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:58 smithi028 ceph-mon[85804]: from='client.54237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:58.909 INFO:teuthology.orchestra.run.smithi028.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:45:59.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:58 smithi066 ceph-mon[68082]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:45:59.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:58 smithi066 ceph-mon[68082]: from='client.54237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:45:59.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T21:45:59.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:59.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:59.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:59.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:59.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:45:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:45:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:45:59.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:45:59 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:00.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:59 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:00.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:59 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:00.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:59 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:00.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:59 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:00.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:45:59 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:00.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:00 smithi028 ceph-mon[85804]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:00.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:00 smithi028 ceph-mon[85804]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:46:00.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:00 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:00.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:00 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:00.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:00 smithi028 ceph-mon[88264]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:00.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:00 smithi028 ceph-mon[88264]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:46:00.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:00 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:00.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:00 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:01.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:00 smithi066 ceph-mon[68082]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:01.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:00 smithi066 ceph-mon[68082]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:46:01.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:00 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:01.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:00 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:01.165 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:46:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:46:00] "GET /metrics HTTP/1.1" 200 37941 "" "Prometheus/2.43.0" 2024-03-21T21:46:01.798 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:46:01.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:01.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:01.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:01.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:01.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:01.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:02.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:02.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:02.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:02.492 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:46:02.492 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (4m) 78s ago 26m 28.9M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:46:02.492 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (3m) 100s ago 25m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:46:02.492 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (3m) 78s ago 25m 53.9M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:46:02.492 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (5m) 100s ago 30m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:46:02.492 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (15m) 78s ago 32m 496M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:46:02.492 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (116s) 78s ago 32m 40.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:46:02.492 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (2m) 100s ago 31m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (89s) 78s ago 31m 20.1M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (4m) 78s ago 26m 15.4M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (4m) 100s ago 26m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (30m) 78s ago 30m 54.4M 2993M 17.2.0 e1d6a67b021e 5d264d90ed85 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (30m) 78s ago 30m 55.6M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (29m) 78s ago 29m 53.5M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (29m) 78s ago 29m 53.6M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (28m) 100s ago 28m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (28m) 100s ago 28m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (28m) 100s ago 28m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (27m) 100s ago 27m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (4m) 100s ago 27m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (25m) 78s ago 25m 78.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:46:02.493 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (25m) 100s ago 25m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:46:02.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:02 smithi028 ceph-mon[88264]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:02.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:02 smithi028 ceph-mon[88264]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:46:02.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:02 smithi028 ceph-mon[85804]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:02 smithi028 ceph-mon[85804]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:46:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:02.922 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:46:02.922 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:46:02.922 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:46:02.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:46:02.922 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:46:02.922 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:46:02.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 5 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:46:02.923 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:46:03.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:02 smithi066 ceph-mon[68082]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:03.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:02 smithi066 ceph-mon[68082]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:46:03.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:03.278 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:46:03.278 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:46:03.279 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:46:03.279 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-03-21T21:46:03.279 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:46:03.279 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "0/8 daemons upgraded", 2024-03-21T21:46:03.279 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T21:46:03.279 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:46:03.279 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:46:03.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='client.54243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:03.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:03.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:03.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/4206751425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:03.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:03.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='client.54243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/4206751425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:03.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:03.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:03.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:03.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:03.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:03.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:03.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:03.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:03 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T21:46:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='client.54243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/4206751425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:04.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:04.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:04.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:04.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:04.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:04.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:04.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:03 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[85804]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[85804]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[85804]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mgr 2024-03-21T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mon 2024-03-21T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all crash 2024-03-21T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T21:46:04.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[85804]: Upgrade: osd.0 is safe to restart 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[85804]: from='client.54267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[88264]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[88264]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[88264]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mgr 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mon 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all crash 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[88264]: Upgrade: osd.0 is safe to restart 2024-03-21T21:46:04.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:04 smithi028 ceph-mon[88264]: from='client.54267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:05.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:04 smithi066 ceph-mon[68082]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:04 smithi066 ceph-mon[68082]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:46:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:04 smithi066 ceph-mon[68082]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:46:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:04 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mgr 2024-03-21T21:46:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:04 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mon 2024-03-21T21:46:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:04 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all crash 2024-03-21T21:46:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:04 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T21:46:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:04 smithi066 ceph-mon[68082]: Upgrade: osd.0 is safe to restart 2024-03-21T21:46:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:04 smithi066 ceph-mon[68082]: from='client.54267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:05.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[88264]: Upgrade: Updating osd.0 2024-03-21T21:46:05.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:05.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T21:46:05.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:05.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[88264]: Deploying daemon osd.0 on smithi028 2024-03-21T21:46:05.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[88264]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:05.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[85804]: Upgrade: Updating osd.0 2024-03-21T21:46:05.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:05.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T21:46:05.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:05.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[85804]: Deploying daemon osd.0 on smithi028 2024-03-21T21:46:05.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:05 smithi028 ceph-mon[85804]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:06.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:05 smithi066 ceph-mon[68082]: Upgrade: Updating osd.0 2024-03-21T21:46:06.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:05 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:06.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:05 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T21:46:06.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:05 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:06.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:05 smithi066 ceph-mon[68082]: Deploying daemon osd.0 on smithi028 2024-03-21T21:46:06.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:05 smithi066 ceph-mon[68082]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:07.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:07 smithi028 ceph-mon[85804]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:07.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:07 smithi028 ceph-mon[88264]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:08.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:07 smithi066 ceph-mon[68082]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:08.447 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:08 smithi028 systemd[1]: Stopping Ceph osd.0 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:46:08.447 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[32437]: 2024-03-21T21:46:08.311+0000 7f7a40f4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:46:08.447 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[32437]: 2024-03-21T21:46:08.311+0000 7f7a40f4d700 -1 osd.0 113 *** Got signal Terminated *** 2024-03-21T21:46:08.447 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:08 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[32437]: 2024-03-21T21:46:08.311+0000 7f7a40f4d700 -1 osd.0 113 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:46:09.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:08 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:09.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:08 smithi028 ceph-mon[88264]: osd.0 marked itself down and dead 2024-03-21T21:46:09.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:08 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:09.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:08 smithi028 ceph-mon[85804]: osd.0 marked itself down and dead 2024-03-21T21:46:09.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:08 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:09.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:08 smithi066 ceph-mon[68082]: osd.0 marked itself down and dead 2024-03-21T21:46:09.547 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:09 smithi028 podman[95277]: 2024-03-21 21:46:09.283703365 +0000 UTC m=+1.066548019 container died 5d264d90ed85a97da3e0d5112ec3b367c2feb303da78ca6d7bf136f3753a6bd8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, version=8, io.buildah.version=1.19.8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.expose-services=, release=754, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-21T21:46:09.820 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:09 smithi028 podman[95277]: 2024-03-21 21:46:09.695744271 +0000 UTC m=+1.478588925 container cleanup 5d264d90ed85a97da3e0d5112ec3b367c2feb303da78ca6d7bf136f3753a6bd8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, distribution-scope=public, release=754, GIT_CLEAN=True, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-21T21:46:09.820 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:09 smithi028 bash[95277]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0 2024-03-21T21:46:10.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:09 smithi028 podman[95294]: 2024-03-21 21:46:09.914112172 +0000 UTC m=+0.622861059 container remove 5d264d90ed85a97da3e0d5112ec3b367c2feb303da78ca6d7bf136f3753a6bd8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, release=754, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, architecture=x86_64, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, name=centos-stream, ceph=True) 2024-03-21T21:46:10.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:09 smithi028 ceph-mon[85804]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:10.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:09 smithi028 ceph-mon[85804]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:46:10.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:09 smithi028 ceph-mon[85804]: osdmap e114: 8 total, 7 up, 8 in 2024-03-21T21:46:10.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:09 smithi028 ceph-mon[88264]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:10.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:09 smithi028 ceph-mon[88264]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:46:10.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:09 smithi028 ceph-mon[88264]: osdmap e114: 8 total, 7 up, 8 in 2024-03-21T21:46:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:09 smithi066 ceph-mon[68082]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:09 smithi066 ceph-mon[68082]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:46:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:09 smithi066 ceph-mon[68082]: osdmap e114: 8 total, 7 up, 8 in 2024-03-21T21:46:10.668 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:10 smithi028 podman[95383]: 2024-03-21 21:46:10.233608805 +0000 UTC m=+0.102271885 container create df6a81469df74569dd0b605d7013ff6ae018e2830345b8240f1e1f9de0eba9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-21T21:46:10.668 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:10 smithi028 podman[95383]: 2024-03-21 21:46:10.149540163 +0000 UTC m=+0.018203255 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:46:10.669 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:10 smithi028 podman[95383]: 2024-03-21 21:46:10.450676555 +0000 UTC m=+0.319339645 container init df6a81469df74569dd0b605d7013ff6ae018e2830345b8240f1e1f9de0eba9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-21T21:46:10.669 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:10 smithi028 podman[95383]: 2024-03-21 21:46:10.454311574 +0000 UTC m=+0.322974654 container start df6a81469df74569dd0b605d7013ff6ae018e2830345b8240f1e1f9de0eba9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS) 2024-03-21T21:46:10.669 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:10 smithi028 podman[95383]: 2024-03-21 21:46:10.667142906 +0000 UTC m=+0.535805985 container attach df6a81469df74569dd0b605d7013ff6ae018e2830345b8240f1e1f9de0eba9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:46:10.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:10 smithi028 ceph-mon[88264]: osdmap e115: 8 total, 7 up, 8 in 2024-03-21T21:46:10.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:10 smithi028 ceph-mon[85804]: osdmap e115: 8 total, 7 up, 8 in 2024-03-21T21:46:10.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:46:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:46:10] "GET /metrics HTTP/1.1" 200 38002 "" "Prometheus/2.43.0" 2024-03-21T21:46:10.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:10 smithi028 podman[95383]: 2024-03-21 21:46:10.75582191 +0000 UTC m=+0.624484989 container died df6a81469df74569dd0b605d7013ff6ae018e2830345b8240f1e1f9de0eba9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:46:11.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:10 smithi066 ceph-mon[68082]: osdmap e115: 8 total, 7 up, 8 in 2024-03-21T21:46:11.830 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:11 smithi028 podman[95409]: 2024-03-21 21:46:11.454950446 +0000 UTC m=+0.693526859 container remove df6a81469df74569dd0b605d7013ff6ae018e2830345b8240f1e1f9de0eba9d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-21T21:46:11.831 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:11 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-21T21:46:11.831 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:11 smithi028 systemd[1]: Stopped Ceph osd.0 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:46:11.831 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:11 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.0.service: Consumed 14.462s CPU time. 2024-03-21T21:46:11.831 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:11 smithi028 systemd[1]: Starting Ceph osd.0 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:46:12.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:11 smithi028 ceph-mon[85804]: pgmap v62: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:46:12.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:11 smithi028 ceph-mon[88264]: pgmap v62: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:46:12.193 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:11 smithi028 podman[95514]: 2024-03-21 21:46:11.816549986 +0000 UTC m=+0.018717176 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:46:12.193 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:11 smithi028 podman[95514]: 2024-03-21 21:46:11.939294552 +0000 UTC m=+0.141461735 container create 7cc030eeaead4c92c7a6ca24fa460090746444098866212853d2c6f7b30f7017 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-21T21:46:12.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:11 smithi066 ceph-mon[68082]: pgmap v62: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:46:12.447 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 podman[95514]: 2024-03-21 21:46:12.192045202 +0000 UTC m=+0.394212394 container init 7cc030eeaead4c92c7a6ca24fa460090746444098866212853d2c6f7b30f7017 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2) 2024-03-21T21:46:12.447 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 podman[95514]: 2024-03-21 21:46:12.19539026 +0000 UTC m=+0.397557444 container start 7cc030eeaead4c92c7a6ca24fa460090746444098866212853d2c6f7b30f7017 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T21:46:12.447 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 podman[95514]: 2024-03-21 21:46:12.283639457 +0000 UTC m=+0.485806644 container attach 7cc030eeaead4c92c7a6ca24fa460090746444098866212853d2c6f7b30f7017 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-21T21:46:12.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:12 smithi028 ceph-mon[88264]: Health check failed: Degraded data redundancy: 65/627 objects degraded (10.367%), 19 pgs degraded (PG_DEGRADED) 2024-03-21T21:46:12.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:12 smithi028 ceph-mon[85804]: Health check failed: Degraded data redundancy: 65/627 objects degraded (10.367%), 19 pgs degraded (PG_DEGRADED) 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate[95529]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate[95529]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 bash[95514]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 bash[95514]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 bash[95514]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 bash[95514]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 bash[95514]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 bash[95514]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 bash[95514]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate[95529]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate[95529]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate[95529]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate[95529]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-21T21:46:12.948 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate[95529]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-21T21:46:12.949 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:12 smithi028 podman[95514]: 2024-03-21 21:46:12.664812756 +0000 UTC m=+0.866979931 container died 7cc030eeaead4c92c7a6ca24fa460090746444098866212853d2c6f7b30f7017 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T21:46:13.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:12 smithi066 ceph-mon[68082]: Health check failed: Degraded data redundancy: 65/627 objects degraded (10.367%), 19 pgs degraded (PG_DEGRADED) 2024-03-21T21:46:13.375 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:13 smithi028 podman[95664]: 2024-03-21 21:46:13.162196097 +0000 UTC m=+0.492210004 container remove 7cc030eeaead4c92c7a6ca24fa460090746444098866212853d2c6f7b30f7017 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-21T21:46:13.683 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:13 smithi028 podman[95729]: 2024-03-21 21:46:13.277155537 +0000 UTC m=+0.018378195 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:46:13.683 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:13 smithi028 podman[95729]: 2024-03-21 21:46:13.379930587 +0000 UTC m=+0.121153285 container create e118d84fcd79b9dc04b014ffd2186f3a855b1dcc82e345f6029358f743cd22da (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0) 2024-03-21T21:46:13.684 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:13 smithi028 podman[95729]: 2024-03-21 21:46:13.57429362 +0000 UTC m=+0.315516290 container init e118d84fcd79b9dc04b014ffd2186f3a855b1dcc82e345f6029358f743cd22da (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T21:46:13.684 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:13 smithi028 podman[95729]: 2024-03-21 21:46:13.577534129 +0000 UTC m=+0.318756789 container start e118d84fcd79b9dc04b014ffd2186f3a855b1dcc82e345f6029358f743cd22da (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T21:46:13.684 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-osd[95748]: -- 172.21.15.28:0/2326001501 <== mon.1 v2:172.21.15.28:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56508dabb680 con 0x56508da85400 2024-03-21T21:46:13.684 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:13 smithi028 bash[95729]: e118d84fcd79b9dc04b014ffd2186f3a855b1dcc82e345f6029358f743cd22da 2024-03-21T21:46:13.947 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:13 smithi028 systemd[1]: Started Ceph osd.0 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:46:13.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[85804]: pgmap v63: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:13.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:13.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:13.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:13.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:13.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[88264]: pgmap v63: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:13.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:13.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:13.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:13.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:14.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:13 smithi066 ceph-mon[68082]: pgmap v63: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:14.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:14.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:14.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:14.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:14.447 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:14 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[95744]: 2024-03-21T21:46:14.119+0000 7f10bb9d77c0 -1 Falling back to public interface 2024-03-21T21:46:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:15 smithi028 ceph-mon[85804]: pgmap v64: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:16.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:15 smithi028 ceph-mon[88264]: pgmap v64: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:16.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:15 smithi066 ceph-mon[68082]: pgmap v64: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:17.112 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[95744]: 2024-03-21T21:46:16.857+0000 7f10bb9d77c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-21T21:46:17.112 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[95744]: 2024-03-21T21:46:16.883+0000 7f10bb9d77c0 -1 osd.0 113 log_to_monitors true 2024-03-21T21:46:17.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[88264]: pgmap v65: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-03-21T21:46:17.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:17.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:17.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[88264]: from='osd.0 [v2:172.21.15.28:6802/692391546,v1:172.21.15.28:6803/692391546]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T21:46:17.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[88264]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T21:46:17.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[85804]: pgmap v65: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-03-21T21:46:17.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:17.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:17.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[85804]: from='osd.0 [v2:172.21.15.28:6802/692391546,v1:172.21.15.28:6803/692391546]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T21:46:17.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-mon[85804]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T21:46:18.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:46:17 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[95744]: 2024-03-21T21:46:17.910+0000 7f10ace2f700 -1 osd.0 113 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:46:18.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:17 smithi066 ceph-mon[68082]: pgmap v65: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-03-21T21:46:18.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:18.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:18.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:17 smithi066 ceph-mon[68082]: from='osd.0 [v2:172.21.15.28:6802/692391546,v1:172.21.15.28:6803/692391546]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T21:46:18.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:17 smithi066 ceph-mon[68082]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[85804]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[85804]: osdmap e116: 8 total, 7 up, 8 in 2024-03-21T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[85804]: from='osd.0 [v2:172.21.15.28:6802/692391546,v1:172.21.15.28:6803/692391546]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[85804]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[88264]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[88264]: osdmap e116: 8 total, 7 up, 8 in 2024-03-21T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[88264]: from='osd.0 [v2:172.21.15.28:6802/692391546,v1:172.21.15.28:6803/692391546]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[88264]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:19.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:18 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:18 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:18 smithi066 ceph-mon[68082]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T21:46:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:18 smithi066 ceph-mon[68082]: osdmap e116: 8 total, 7 up, 8 in 2024-03-21T21:46:19.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:18 smithi066 ceph-mon[68082]: from='osd.0 [v2:172.21.15.28:6802/692391546,v1:172.21.15.28:6803/692391546]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:19.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:18 smithi066 ceph-mon[68082]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:19.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:18 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:19.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:18 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:20.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[88264]: pgmap v67: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 458 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:20.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[88264]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:46:20.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[88264]: osd.0 [v2:172.21.15.28:6802/692391546,v1:172.21.15.28:6803/692391546] boot 2024-03-21T21:46:20.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[88264]: osdmap e117: 8 total, 8 up, 8 in 2024-03-21T21:46:20.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:46:20.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[88264]: osdmap e118: 8 total, 8 up, 8 in 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[85804]: pgmap v67: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 458 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[85804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[85804]: osd.0 [v2:172.21.15.28:6802/692391546,v1:172.21.15.28:6803/692391546] boot 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[85804]: osdmap e117: 8 total, 8 up, 8 in 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:19 smithi028 ceph-mon[85804]: osdmap e118: 8 total, 8 up, 8 in 2024-03-21T21:46:20.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:19 smithi066 ceph-mon[68082]: pgmap v67: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 458 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:20.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:19 smithi066 ceph-mon[68082]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:46:20.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:19 smithi066 ceph-mon[68082]: osd.0 [v2:172.21.15.28:6802/692391546,v1:172.21.15.28:6803/692391546] boot 2024-03-21T21:46:20.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:19 smithi066 ceph-mon[68082]: osdmap e117: 8 total, 8 up, 8 in 2024-03-21T21:46:20.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:19 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T21:46:20.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:19 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:20.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:19 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:20.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:19 smithi066 ceph-mon[68082]: osdmap e118: 8 total, 8 up, 8 in 2024-03-21T21:46:21.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:46:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:46:20] "GET /metrics HTTP/1.1" 200 38142 "" "Prometheus/2.43.0" 2024-03-21T21:46:22.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:22 smithi028 ceph-mon[88264]: pgmap v70: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:22.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:22 smithi028 ceph-mon[85804]: pgmap v70: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:23.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:22 smithi066 ceph-mon[68082]: pgmap v70: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T21:46:23.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:46:22 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:46:22.987+0000 7f15e6bd1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-21T21:46:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: pgmap v71: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 28/627 objects degraded (4.466%) 2024-03-21T21:46:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: pgmap v71: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 28/627 objects degraded (4.466%) 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-21T21:46:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:23 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: pgmap v71: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 28/627 objects degraded (4.466%) 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:24.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:24.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:24.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-21T21:46:24.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:23 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T21:46:25.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:24 smithi028 ceph-mon[88264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-03-21T21:46:25.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:24 smithi028 ceph-mon[88264]: Cluster is now healthy 2024-03-21T21:46:25.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:24 smithi028 ceph-mon[85804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-03-21T21:46:25.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:24 smithi028 ceph-mon[85804]: Cluster is now healthy 2024-03-21T21:46:25.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:24 smithi066 ceph-mon[68082]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-03-21T21:46:25.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:24 smithi066 ceph-mon[68082]: Cluster is now healthy 2024-03-21T21:46:26.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:25 smithi028 ceph-mon[88264]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-03-21T21:46:26.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:25 smithi028 ceph-mon[85804]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-03-21T21:46:26.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:25 smithi066 ceph-mon[68082]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-03-21T21:46:28.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:27 smithi028 ceph-mon[88264]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:46:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:27 smithi028 ceph-mon[85804]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:46:28.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:27 smithi066 ceph-mon[68082]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:46:29.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:28 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:29.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:28 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:29.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:28 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:30.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:29 smithi028 ceph-mon[88264]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:46:30.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:29 smithi028 ceph-mon[85804]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:46:30.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:29 smithi066 ceph-mon[68082]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:46:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:46:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:46:30] "GET /metrics HTTP/1.1" 200 38142 "" "Prometheus/2.43.0" 2024-03-21T21:46:32.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:31 smithi028 ceph-mon[85804]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-21T21:46:32.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:31 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:32.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:31 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:32.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:31 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:32.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:31 smithi028 ceph-mon[88264]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-21T21:46:32.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:31 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:32.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:31 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:32.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:31 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:32.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:31 smithi066 ceph-mon[68082]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-21T21:46:32.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:31 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:32.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:31 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:32.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:31 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:33.646 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:46:34.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:33 smithi028 ceph-mon[88264]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:34.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:33 smithi028 ceph-mon[88264]: from='client.64216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:34.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:33 smithi028 ceph-mon[85804]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:34.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:33 smithi028 ceph-mon[85804]: from='client.64216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:34.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:33 smithi066 ceph-mon[68082]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:34.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:33 smithi066 ceph-mon[68082]: from='client.64216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (4m) 16s ago 26m 28.9M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (4m) 2m ago 26m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (3m) 16s ago 25m 54.1M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (5m) 2m ago 31m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (15m) 16s ago 33m 504M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (2m) 16s ago 33m 45.3M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (3m) 2m ago 31m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (2m) 16s ago 31m 37.3M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (5m) 16s ago 27m 15.0M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (5m) 2m ago 27m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (21s) 16s ago 31m 12.0M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (30m) 16s ago 30m 55.4M 2993M 17.2.0 e1d6a67b021e 1b6513d118f4 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (30m) 16s ago 30m 54.2M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:46:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (29m) 16s ago 29m 54.3M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:46:34.342 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (29m) 2m ago 29m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:46:34.342 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (29m) 2m ago 29m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:46:34.342 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (28m) 2m ago 28m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:46:34.342 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (28m) 2m ago 28m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:46:34.342 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (4m) 2m ago 27m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:46:34.342 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (25m) 16s ago 25m 78.4M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:46:34.342 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (25m) 2m ago 25m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 1 2024-03-21T21:46:34.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:46:34.750 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:46:34.750 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:46:34.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:46:34.750 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:46:34.750 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-21T21:46:34.750 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 6 2024-03-21T21:46:34.750 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:46:34.750 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:46:35.096 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:46:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:46:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:46:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-03-21T21:46:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:46:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "1/8 daemons upgraded", 2024-03-21T21:46:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T21:46:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:46:35.097 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:46:35.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:34 smithi028 ceph-mon[88264]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:35.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:34 smithi028 ceph-mon[88264]: from='client.54291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:35.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:34 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/3357913074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:35.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:34 smithi028 ceph-mon[85804]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:35.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:34 smithi028 ceph-mon[85804]: from='client.54291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:35.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:34 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/3357913074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:35.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:34 smithi066 ceph-mon[68082]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:35.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:34 smithi066 ceph-mon[68082]: from='client.54291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:35.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:34 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/3357913074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:36.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:35 smithi028 ceph-mon[88264]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:36.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:35 smithi028 ceph-mon[88264]: from='client.54303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:36.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:35 smithi028 ceph-mon[85804]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:36.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:35 smithi028 ceph-mon[85804]: from='client.54303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:36.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:35 smithi066 ceph-mon[68082]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:36.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:35 smithi066 ceph-mon[68082]: from='client.54303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:46:38.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:37 smithi028 ceph-mon[88264]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:38.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:37 smithi028 ceph-mon[85804]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:38.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:37 smithi066 ceph-mon[68082]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:39.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:38 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:39.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:38 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:39.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:38 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:39.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:38 smithi028 ceph-mon[88264]: Upgrade: osd.1 is safe to restart 2024-03-21T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:38 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:38 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:38 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:38 smithi028 ceph-mon[85804]: Upgrade: osd.1 is safe to restart 2024-03-21T21:46:39.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:38 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:39.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:38 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:39.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:38 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T21:46:39.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:38 smithi066 ceph-mon[68082]: Upgrade: osd.1 is safe to restart 2024-03-21T21:46:40.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:39 smithi028 ceph-mon[88264]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:40.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:40.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T21:46:40.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:40.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:39 smithi028 ceph-mon[85804]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:40.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:40.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T21:46:40.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:40.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:39 smithi066 ceph-mon[68082]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:40.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:40.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T21:46:40.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:41.129 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:46:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:46:40] "GET /metrics HTTP/1.1" 200 38163 "" "Prometheus/2.43.0" 2024-03-21T21:46:41.130 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:40 smithi028 ceph-mon[88264]: Upgrade: Updating osd.1 2024-03-21T21:46:41.130 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:40 smithi028 ceph-mon[88264]: Deploying daemon osd.1 on smithi028 2024-03-21T21:46:41.130 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:40 smithi028 ceph-mon[85804]: Upgrade: Updating osd.1 2024-03-21T21:46:41.130 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:40 smithi028 ceph-mon[85804]: Deploying daemon osd.1 on smithi028 2024-03-21T21:46:41.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:40 smithi066 ceph-mon[68082]: Upgrade: Updating osd.1 2024-03-21T21:46:41.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:40 smithi066 ceph-mon[68082]: Deploying daemon osd.1 on smithi028 2024-03-21T21:46:42.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:41 smithi028 ceph-mon[88264]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:42.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:41 smithi028 ceph-mon[85804]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:42.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:41 smithi066 ceph-mon[68082]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:46:43.367 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:43 smithi028 systemd[1]: Stopping Ceph osd.1 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:46:43.697 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[35393]: 2024-03-21T21:46:43.364+0000 7f4d4d0ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:46:43.697 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[35393]: 2024-03-21T21:46:43.364+0000 7f4d4d0ff700 -1 osd.1 118 *** Got signal Terminated *** 2024-03-21T21:46:43.697 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[35393]: 2024-03-21T21:46:43.364+0000 7f4d4d0ff700 -1 osd.1 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:46:44.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:43 smithi028 ceph-mon[85804]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:44.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:43 smithi028 ceph-mon[85804]: osd.1 marked itself down and dead 2024-03-21T21:46:44.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:43 smithi028 ceph-mon[88264]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:44.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:43 smithi028 ceph-mon[88264]: osd.1 marked itself down and dead 2024-03-21T21:46:44.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:43 smithi066 ceph-mon[68082]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:46:44.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:43 smithi066 ceph-mon[68082]: osd.1 marked itself down and dead 2024-03-21T21:46:44.634 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:44 smithi028 podman[100063]: 2024-03-21 21:46:44.328028289 +0000 UTC m=+1.080293017 container died 1b6513d118f45357d5f1b12b302446958e774ea6ebe37338d0066ddfa1c1f868 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1, name=centos-stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, release=754, build-date=2022-05-03T08:36:31.336870, ceph=True) 2024-03-21T21:46:44.928 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:44 smithi028 podman[100063]: 2024-03-21 21:46:44.723507343 +0000 UTC m=+1.475772081 container cleanup 1b6513d118f45357d5f1b12b302446958e774ea6ebe37338d0066ddfa1c1f868 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., vcs-type=git, com.redhat.component=centos-stream-container, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , RELEASE=HEAD, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_CLEAN=True, distribution-scope=public, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-21T21:46:44.928 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:44 smithi028 bash[100063]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1 2024-03-21T21:46:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:44 smithi028 ceph-mon[88264]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:46:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:44 smithi028 ceph-mon[88264]: osdmap e119: 8 total, 7 up, 8 in 2024-03-21T21:46:45.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:44 smithi028 ceph-mon[85804]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:46:45.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:44 smithi028 ceph-mon[85804]: osdmap e119: 8 total, 7 up, 8 in 2024-03-21T21:46:45.198 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:44 smithi028 podman[100081]: 2024-03-21 21:46:44.967388814 +0000 UTC m=+0.638803978 container remove 1b6513d118f45357d5f1b12b302446958e774ea6ebe37338d0066ddfa1c1f868 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1, vcs-type=git, io.buildah.version=1.19.8, distribution-scope=public, architecture=x86_64, vendor=Red Hat, Inc., RELEASE=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-21T21:46:45.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:44 smithi066 ceph-mon[68082]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:46:45.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:44 smithi066 ceph-mon[68082]: osdmap e119: 8 total, 7 up, 8 in 2024-03-21T21:46:45.646 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:45 smithi028 podman[100175]: 2024-03-21 21:46:45.193511495 +0000 UTC m=+0.019017304 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:46:45.646 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:45 smithi028 podman[100175]: 2024-03-21 21:46:45.300381129 +0000 UTC m=+0.125886919 container create 9f69ed1d56ddd05451d0bde10a8c5ac67ed66fb499b68a262f426fb68fe02caa (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-21T21:46:45.646 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:45 smithi028 podman[100175]: 2024-03-21 21:46:45.536380647 +0000 UTC m=+0.361886462 container init 9f69ed1d56ddd05451d0bde10a8c5ac67ed66fb499b68a262f426fb68fe02caa (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T21:46:45.646 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:45 smithi028 podman[100175]: 2024-03-21 21:46:45.539632917 +0000 UTC m=+0.365138715 container start 9f69ed1d56ddd05451d0bde10a8c5ac67ed66fb499b68a262f426fb68fe02caa (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-21T21:46:45.646 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:45 smithi028 podman[100175]: 2024-03-21 21:46:45.644826469 +0000 UTC m=+0.470332259 container attach 9f69ed1d56ddd05451d0bde10a8c5ac67ed66fb499b68a262f426fb68fe02caa (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-21T21:46:45.941 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:45 smithi028 podman[100175]: 2024-03-21 21:46:45.838423829 +0000 UTC m=+0.663929628 container died 9f69ed1d56ddd05451d0bde10a8c5ac67ed66fb499b68a262f426fb68fe02caa (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-21T21:46:46.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:45 smithi028 ceph-mon[85804]: pgmap v83: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:46:46.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:45 smithi028 ceph-mon[85804]: osdmap e120: 8 total, 7 up, 8 in 2024-03-21T21:46:46.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:45 smithi028 ceph-mon[88264]: pgmap v83: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:46:46.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:45 smithi028 ceph-mon[88264]: osdmap e120: 8 total, 7 up, 8 in 2024-03-21T21:46:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:45 smithi066 ceph-mon[68082]: pgmap v83: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:46:46.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:45 smithi066 ceph-mon[68082]: osdmap e120: 8 total, 7 up, 8 in 2024-03-21T21:46:46.697 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:46 smithi028 podman[100201]: 2024-03-21 21:46:46.423349142 +0000 UTC m=+0.579883207 container remove 9f69ed1d56ddd05451d0bde10a8c5ac67ed66fb499b68a262f426fb68fe02caa (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-21T21:46:46.697 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:46 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-21T21:46:46.698 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:46 smithi028 systemd[1]: Stopped Ceph osd.1 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:46:46.698 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:46 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.1.service: Consumed 15.348s CPU time. 2024-03-21T21:46:46.698 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:46 smithi028 systemd[1]: Starting Ceph osd.1 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:46:47.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:46 smithi028 podman[100308]: 2024-03-21 21:46:46.758143605 +0000 UTC m=+0.017758307 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:46:47.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:46 smithi028 podman[100308]: 2024-03-21 21:46:46.88268903 +0000 UTC m=+0.142303729 container create 0722e05d3c74e3b79c8a9870635ab4d757b039539e8f8bd1f973aa789e53807f (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:46:47.447 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 podman[100308]: 2024-03-21 21:46:47.126739463 +0000 UTC m=+0.386354174 container init 0722e05d3c74e3b79c8a9870635ab4d757b039539e8f8bd1f973aa789e53807f (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-21T21:46:47.447 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 podman[100308]: 2024-03-21 21:46:47.130208856 +0000 UTC m=+0.389823559 container start 0722e05d3c74e3b79c8a9870635ab4d757b039539e8f8bd1f973aa789e53807f (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-21T21:46:47.447 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 podman[100308]: 2024-03-21 21:46:47.206956521 +0000 UTC m=+0.466571221 container attach 0722e05d3c74e3b79c8a9870635ab4d757b039539e8f8bd1f973aa789e53807f (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-21T21:46:47.828 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate[100322]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-21T21:46:47.828 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate[100322]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-21T21:46:47.828 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 bash[100308]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-21T21:46:47.828 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 bash[100308]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 bash[100308]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 bash[100308]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 bash[100308]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 bash[100308]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 bash[100308]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate[100322]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate[100322]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate[100322]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate[100322]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate[100322]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-21T21:46:47.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:47 smithi028 podman[100308]: 2024-03-21 21:46:47.59702714 +0000 UTC m=+0.856641842 container died 0722e05d3c74e3b79c8a9870635ab4d757b039539e8f8bd1f973aa789e53807f (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-21T21:46:48.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-mon[88264]: pgmap v85: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:46:48.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:48.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:48.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-mon[85804]: pgmap v85: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:46:48.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:48.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:48.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:47 smithi066 ceph-mon[68082]: pgmap v85: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:46:48.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:48.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:46:48.447 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:48 smithi028 podman[100453]: 2024-03-21 21:46:48.147436983 +0000 UTC m=+0.545670816 container remove 0722e05d3c74e3b79c8a9870635ab4d757b039539e8f8bd1f973aa789e53807f (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-activate, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-21T21:46:48.448 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:48 smithi028 podman[100520]: 2024-03-21 21:46:48.264142884 +0000 UTC m=+0.017979692 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:46:48.448 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:48 smithi028 podman[100520]: 2024-03-21 21:46:48.365112493 +0000 UTC m=+0.118949293 container create 63310efb44c3a339ebc3ddb92bba5787ec9f2dd5526e14fb48d909b8d1b9f4f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-21T21:46:48.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:48 smithi028 podman[100520]: 2024-03-21 21:46:48.594108469 +0000 UTC m=+0.347945262 container init 63310efb44c3a339ebc3ddb92bba5787ec9f2dd5526e14fb48d909b8d1b9f4f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-21T21:46:48.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:48 smithi028 podman[100520]: 2024-03-21 21:46:48.59720644 +0000 UTC m=+0.351043235 container start 63310efb44c3a339ebc3ddb92bba5787ec9f2dd5526e14fb48d909b8d1b9f4f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:46:48.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-osd[100540]: -- 172.21.15.28:0/2960799429 <== mon.1 v2:172.21.15.28:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5652e3a5f680 con 0x5652e3a28c00 2024-03-21T21:46:48.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:48 smithi028 bash[100520]: 63310efb44c3a339ebc3ddb92bba5787ec9f2dd5526e14fb48d909b8d1b9f4f3 2024-03-21T21:46:48.829 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:48 smithi028 systemd[1]: Started Ceph osd.1 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:46:49.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:49.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:49.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:49.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:49.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:49.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:49.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:49.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:49.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:49.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:49.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:48 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:49.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:49.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:49.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:49.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:49.447 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:49 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[100536]: 2024-03-21T21:46:49.137+0000 7fd6bea187c0 -1 Falling back to public interface 2024-03-21T21:46:50.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:49 smithi028 ceph-mon[88264]: pgmap v86: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:50.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:49 smithi028 ceph-mon[88264]: Health check failed: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T21:46:50.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:49 smithi028 ceph-mon[85804]: pgmap v86: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:50.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:49 smithi028 ceph-mon[85804]: Health check failed: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T21:46:50.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:49 smithi066 ceph-mon[68082]: pgmap v86: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:50.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:49 smithi066 ceph-mon[68082]: Health check failed: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T21:46:51.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:46:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:46:50] "GET /metrics HTTP/1.1" 200 38157 "" "Prometheus/2.43.0" 2024-03-21T21:46:51.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:51 smithi028 ceph-mon[85804]: pgmap v87: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:51.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:51 smithi028 ceph-mon[88264]: pgmap v87: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:51.947 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:51 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[100536]: 2024-03-21T21:46:51.855+0000 7fd6bea187c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-21T21:46:51.947 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:51 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[100536]: 2024-03-21T21:46:51.886+0000 7fd6bea187c0 -1 osd.1 118 log_to_monitors true 2024-03-21T21:46:52.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:51 smithi066 ceph-mon[68082]: pgmap v87: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:53.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:52 smithi028 ceph-mon[88264]: from='osd.1 [v2:172.21.15.28:6810/3415545707,v1:172.21.15.28:6811/3415545707]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T21:46:53.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:52 smithi028 ceph-mon[88264]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T21:46:53.111 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:46:52 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[100536]: 2024-03-21T21:46:52.861+0000 7fd6afe70700 -1 osd.1 118 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:46:53.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:52 smithi028 ceph-mon[85804]: from='osd.1 [v2:172.21.15.28:6810/3415545707,v1:172.21.15.28:6811/3415545707]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T21:46:53.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:52 smithi028 ceph-mon[85804]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T21:46:53.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:52 smithi066 ceph-mon[68082]: from='osd.1 [v2:172.21.15.28:6810/3415545707,v1:172.21.15.28:6811/3415545707]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T21:46:53.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:52 smithi066 ceph-mon[68082]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T21:46:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[88264]: pgmap v88: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[88264]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T21:46:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[88264]: osdmap e121: 8 total, 7 up, 8 in 2024-03-21T21:46:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[88264]: from='osd.1 [v2:172.21.15.28:6810/3415545707,v1:172.21.15.28:6811/3415545707]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[88264]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[85804]: pgmap v88: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[85804]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[85804]: osdmap e121: 8 total, 7 up, 8 in 2024-03-21T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[85804]: from='osd.1 [v2:172.21.15.28:6810/3415545707,v1:172.21.15.28:6811/3415545707]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[85804]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:54.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:54.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:53 smithi066 ceph-mon[68082]: pgmap v88: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:54.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:53 smithi066 ceph-mon[68082]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T21:46:54.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:53 smithi066 ceph-mon[68082]: osdmap e121: 8 total, 7 up, 8 in 2024-03-21T21:46:54.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:53 smithi066 ceph-mon[68082]: from='osd.1 [v2:172.21.15.28:6810/3415545707,v1:172.21.15.28:6811/3415545707]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:54.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:53 smithi066 ceph-mon[68082]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:46:54.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:54.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:55.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[88264]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:46:55.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[88264]: osd.1 [v2:172.21.15.28:6810/3415545707,v1:172.21.15.28:6811/3415545707] boot 2024-03-21T21:46:55.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[88264]: osdmap e122: 8 total, 8 up, 8 in 2024-03-21T21:46:55.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:46:55.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:55.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:55.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[88264]: osdmap e123: 8 total, 8 up, 8 in 2024-03-21T21:46:55.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[85804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:46:55.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[85804]: osd.1 [v2:172.21.15.28:6810/3415545707,v1:172.21.15.28:6811/3415545707] boot 2024-03-21T21:46:55.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[85804]: osdmap e122: 8 total, 8 up, 8 in 2024-03-21T21:46:55.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:46:55.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:55.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:55.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:54 smithi028 ceph-mon[85804]: osdmap e123: 8 total, 8 up, 8 in 2024-03-21T21:46:55.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:54 smithi066 ceph-mon[68082]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:46:55.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:54 smithi066 ceph-mon[68082]: osd.1 [v2:172.21.15.28:6810/3415545707,v1:172.21.15.28:6811/3415545707] boot 2024-03-21T21:46:55.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:54 smithi066 ceph-mon[68082]: osdmap e122: 8 total, 8 up, 8 in 2024-03-21T21:46:55.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:54 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T21:46:55.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:54 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:55.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:54 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:55.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:54 smithi066 ceph-mon[68082]: osdmap e123: 8 total, 8 up, 8 in 2024-03-21T21:46:56.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:56 smithi066 ceph-mon[68082]: pgmap v91: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:56.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:56 smithi028 ceph-mon[85804]: pgmap v91: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:56.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:56 smithi028 ceph-mon[88264]: pgmap v91: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-03-21T21:46:58.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: pgmap v93: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/627 objects degraded (13.397%) 2024-03-21T21:46:58.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:58.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:46:58.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:58.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:58 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: pgmap v93: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/627 objects degraded (13.397%) 2024-03-21T21:46:58.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:58.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:58.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:58.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:46:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: pgmap v93: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/627 objects degraded (13.397%) 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:46:58.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:46:58.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:46:58.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:46:58.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:46:58.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:46:58.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:46:58.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:46:58.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:46:58.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:58.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:46:58.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:58 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:46:59.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:59 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:59.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:59 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:46:59.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:59 smithi028 ceph-mon[85804]: Upgrade: Finalizing container_image settings 2024-03-21T21:46:59.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:46:59 smithi028 ceph-mon[85804]: Upgrade: Complete! 2024-03-21T21:46:59.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:59 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:46:59.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:59 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:46:59.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:59 smithi028 ceph-mon[88264]: Upgrade: Finalizing container_image settings 2024-03-21T21:46:59.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:46:59 smithi028 ceph-mon[88264]: Upgrade: Complete! 2024-03-21T21:47:00.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:59 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:00.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:59 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:47:00.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:59 smithi066 ceph-mon[68082]: Upgrade: Finalizing container_image settings 2024-03-21T21:47:00.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:46:59 smithi066 ceph-mon[68082]: Upgrade: Complete! 2024-03-21T21:47:00.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:00 smithi028 ceph-mon[88264]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-21T21:47:00.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:00 smithi028 ceph-mon[88264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded) 2024-03-21T21:47:00.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:00 smithi028 ceph-mon[88264]: Cluster is now healthy 2024-03-21T21:47:00.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:00 smithi028 ceph-mon[85804]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-21T21:47:00.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:00 smithi028 ceph-mon[85804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded) 2024-03-21T21:47:00.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:00 smithi028 ceph-mon[85804]: Cluster is now healthy 2024-03-21T21:47:01.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:00 smithi066 ceph-mon[68082]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-21T21:47:01.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:00 smithi066 ceph-mon[68082]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded) 2024-03-21T21:47:01.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:00 smithi066 ceph-mon[68082]: Cluster is now healthy 2024-03-21T21:47:01.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:47:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:47:00] "GET /metrics HTTP/1.1" 200 38157 "" "Prometheus/2.43.0" 2024-03-21T21:47:02.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:02 smithi028 ceph-mon[85804]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-21T21:47:02.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:02.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:02.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:02.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:02 smithi028 ceph-mon[88264]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-21T21:47:02.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:02.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:02.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:03.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:02 smithi066 ceph-mon[68082]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-21T21:47:03.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:03.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:03.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:04.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:04 smithi028 ceph-mon[85804]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-21T21:47:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:04 smithi028 ceph-mon[88264]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-21T21:47:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:04 smithi066 ceph-mon[68082]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-21T21:47:06.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:47:06.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:06 smithi028 ceph-mon[88264]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-21T21:47:06.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:06 smithi028 ceph-mon[88264]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:06.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:06 smithi028 ceph-mon[85804]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-21T21:47:06.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:06 smithi028 ceph-mon[85804]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:06 smithi066 ceph-mon[68082]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-21T21:47:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:06 smithi066 ceph-mon[68082]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:08.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:07 smithi028 ceph-mon[85804]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T21:47:08.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:07 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:08.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:07 smithi028 ceph-mon[88264]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T21:47:08.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:07 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:08.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:07 smithi066 ceph-mon[68082]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T21:47:08.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:07 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:08.274 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:47:08.274 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (5m) 14s ago 27m 27.4M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:47:08.274 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (4m) 2m ago 26m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:47:08.274 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (4m) 14s ago 26m 55.9M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:47:08.274 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (6m) 2m ago 31m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:47:08.274 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (16m) 14s ago 33m 506M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:47:08.274 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (3m) 14s ago 33m 46.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (3m) 2m ago 32m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (2m) 14s ago 32m 38.2M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (5m) 14s ago 28m 15.1M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (5m) 2m ago 27m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (54s) 14s ago 31m 43.8M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (20s) 14s ago 31m 12.4M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (30m) 14s ago 30m 54.3M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (30m) 14s ago 30m 54.9M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (30m) 2m ago 30m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (29m) 2m ago 29m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (29m) 2m ago 29m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (28m) 2m ago 28m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (5m) 2m ago 28m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:47:08.275 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (26m) 14s ago 26m 78.6M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:47:08.276 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (26m) 2m ago 26m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:47:09.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:08 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:09.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:08 smithi028 ceph-mon[85804]: from='client.54321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:09.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:08 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:09.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:08 smithi028 ceph-mon[88264]: from='client.54321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:09.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-21T21:47:09.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:08 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:09.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:08 smithi066 ceph-mon[68082]: from='client.54321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:10.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:09 smithi028 ceph-mon[85804]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-21T21:47:10.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:09 smithi028 ceph-mon[88264]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-21T21:47:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:09 smithi066 ceph-mon[68082]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-21T21:47:11.025 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:47:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:47:10] "GET /metrics HTTP/1.1" 200 38170 "" "Prometheus/2.43.0" 2024-03-21T21:47:11.302 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:47:11.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-03-21T21:47:12.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:11 smithi028 ceph-mon[88264]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:12.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:11 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/4040467934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:11 smithi028 ceph-mon[85804]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:12.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:11 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/4040467934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:12.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:11 smithi066 ceph-mon[68082]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:12.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:11 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/4040467934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:14.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:13 smithi028 ceph-mon[85804]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:14.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:13 smithi028 ceph-mon[88264]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:14.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:13 smithi066 ceph-mon[68082]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:15.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:14 smithi028 ceph-mon[85804]: from='client.54333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:15.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:14 smithi028 ceph-mon[88264]: from='client.54333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:15.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:14 smithi066 ceph-mon[68082]: from='client.54333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:16.110 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:47:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:15 smithi028 ceph-mon[85804]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:16.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:15 smithi028 ceph-mon[88264]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:16.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:15 smithi066 ceph-mon[68082]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:16.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade status' 2024-03-21T21:47:17.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:17.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:17.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:18.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:17 smithi028 ceph-mon[85804]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:18.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:17 smithi028 ceph-mon[88264]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:18.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:17 smithi066 ceph-mon[68082]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:18.611 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:47:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": null, 2024-03-21T21:47:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": false, 2024-03-21T21:47:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "", 2024-03-21T21:47:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:47:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": null, 2024-03-21T21:47:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-03-21T21:47:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:47:18.612 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:47:19.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:18 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:19.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:18 smithi028 ceph-mon[88264]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:18 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:19.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:18 smithi028 ceph-mon[85804]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:19.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:18 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:19.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:18 smithi066 ceph-mon[68082]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:19.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:47:20.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:19 smithi028 ceph-mon[85804]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:20.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:19 smithi028 ceph-mon[88264]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:20.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:19 smithi066 ceph-mon[68082]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:20.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:47:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:47:20] "GET /metrics HTTP/1.1" 200 38165 "" "Prometheus/2.43.0" 2024-03-21T21:47:21.521 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-03-21T21:47:22.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:21 smithi028 ceph-mon[88264]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:22.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:21 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/1584910529' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:47:22.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:21 smithi028 ceph-mon[85804]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:22.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:21 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/1584910529' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:47:22.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-03-21T21:47:22.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:21 smithi066 ceph-mon[68082]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:22.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:21 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/1584910529' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:47:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:23 smithi028 ceph-mon[85804]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:23 smithi028 ceph-mon[88264]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:24.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:23 smithi066 ceph-mon[68082]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:25.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:24 smithi028 ceph-mon[88264]: from='client.54351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:25.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:24 smithi028 ceph-mon[85804]: from='client.54351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:25.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:24 smithi066 ceph-mon[68082]: from='client.54351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:26.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:25 smithi028 ceph-mon[88264]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:25 smithi028 ceph-mon[85804]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:26.231 INFO:teuthology.orchestra.run.smithi028.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:47:26.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:25 smithi066 ceph-mon[68082]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:26.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T21:47:27.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[85804]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:47:27.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:27.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:27.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:27.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[88264]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:47:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:27 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:27.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:27 smithi066 ceph-mon[68082]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:47:27.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:27.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:27.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:27.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:27.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:28.423 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:28 smithi028 ceph-mon[88264]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:28.423 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:28 smithi028 ceph-mon[88264]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:47:28.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:28 smithi028 ceph-mon[85804]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:28.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:28 smithi028 ceph-mon[85804]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:47:28.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:28 smithi066 ceph-mon[68082]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:28.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:28 smithi066 ceph-mon[68082]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:47:28.797 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:47:29.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:29 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:29.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:29 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:29.479 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:47:29.479 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (5m) 36s ago 27m 27.4M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:47:29.479 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (5m) 3m ago 27m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:47:29.479 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (4m) 36s ago 26m 55.9M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:47:29.479 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (6m) 3m ago 32m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:47:29.479 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (16m) 36s ago 33m 506M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (3m) 36s ago 34m 46.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (4m) 3m ago 32m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (2m) 36s ago 32m 38.2M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (6m) 36s ago 28m 15.1M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (6m) 3m ago 28m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (76s) 36s ago 31m 43.8M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (41s) 36s ago 31m 12.4M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (31m) 36s ago 31m 54.3M 2993M 17.2.0 e1d6a67b021e 2b223dfc5f58 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (30m) 36s ago 30m 54.9M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (30m) 3m ago 30m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (29m) 3m ago 29m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (29m) 3m ago 29m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (29m) 3m ago 29m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (5m) 3m ago 28m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:47:29.480 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (26m) 36s ago 26m 78.6M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:47:29.481 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (26m) 3m ago 26m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:47:29.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:29 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:47:29.898 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:47:29.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:47:29.899 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:47:29.899 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-21T21:47:29.899 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 7 2024-03-21T21:47:29.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:47:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:47:30.248 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:47:30.248 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:47:30.248 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:47:30.248 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-03-21T21:47:30.248 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:47:30.248 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "2/8 daemons upgraded", 2024-03-21T21:47:30.248 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T21:47:30.248 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:47:30.248 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:47:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='client.54369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/1793999030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='client.54369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T21:47:30.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/1793999030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='client.54369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:30.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:30.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T21:47:30.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:30 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/1793999030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:47:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:47:30] "GET /metrics HTTP/1.1" 200 38165 "" "Prometheus/2.43.0" 2024-03-21T21:47:31.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[88264]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:47:31.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[88264]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mgr 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mon 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all crash 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[88264]: Upgrade: osd.2 is safe to restart 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[88264]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[85804]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[85804]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mgr 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mon 2024-03-21T21:47:31.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all crash 2024-03-21T21:47:31.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T21:47:31.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[85804]: Upgrade: osd.2 is safe to restart 2024-03-21T21:47:31.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:31 smithi028 ceph-mon[85804]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:31.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:31 smithi066 ceph-mon[68082]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:47:31.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:31 smithi066 ceph-mon[68082]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:47:31.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:31 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mgr 2024-03-21T21:47:31.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:31 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mon 2024-03-21T21:47:31.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:31 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all crash 2024-03-21T21:47:31.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:31 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T21:47:31.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:31 smithi066 ceph-mon[68082]: Upgrade: osd.2 is safe to restart 2024-03-21T21:47:31.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:31 smithi066 ceph-mon[68082]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:47:32.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[88264]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:32.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:32.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T21:47:32.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:32.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:32.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[85804]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:32.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:32.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T21:47:32.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:32.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:32.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:32 smithi066 ceph-mon[68082]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:32.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:32.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T21:47:32.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:32.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:33.556 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:33 smithi028 ceph-mon[88264]: Upgrade: Updating osd.2 2024-03-21T21:47:33.556 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:33 smithi028 ceph-mon[88264]: Deploying daemon osd.2 on smithi028 2024-03-21T21:47:33.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:33 smithi028 ceph-mon[85804]: Upgrade: Updating osd.2 2024-03-21T21:47:33.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:33 smithi028 ceph-mon[85804]: Deploying daemon osd.2 on smithi028 2024-03-21T21:47:33.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:33 smithi066 ceph-mon[68082]: Upgrade: Updating osd.2 2024-03-21T21:47:33.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:33 smithi066 ceph-mon[68082]: Deploying daemon osd.2 on smithi028 2024-03-21T21:47:34.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:34 smithi028 ceph-mon[85804]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:34.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:34 smithi028 ceph-mon[88264]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:34.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:34 smithi066 ceph-mon[68082]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:35.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:35 smithi028 systemd[1]: Stopping Ceph osd.2 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:47:35.620 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:35 smithi028 ceph-mon[88264]: osd.2 marked itself down and dead 2024-03-21T21:47:35.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:35 smithi028 ceph-mon[85804]: osd.2 marked itself down and dead 2024-03-21T21:47:35.620 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[38316]: 2024-03-21T21:47:35.249+0000 7fc382c7b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:47:35.621 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[38316]: 2024-03-21T21:47:35.249+0000 7fc382c7b700 -1 osd.2 123 *** Got signal Terminated *** 2024-03-21T21:47:35.621 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[38316]: 2024-03-21T21:47:35.249+0000 7fc382c7b700 -1 osd.2 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:47:35.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:35 smithi066 ceph-mon[68082]: osd.2 marked itself down and dead 2024-03-21T21:47:35.876 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:35 smithi028 podman[106708]: 2024-03-21 21:47:35.619003763 +0000 UTC m=+0.464115719 container died 2b223dfc5f5843b5bb880054b7d94d843bcef3fa06a259480e4936790e1dab05 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2, architecture=x86_64, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, version=8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, name=centos-stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-21T21:47:36.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:35 smithi028 podman[106708]: 2024-03-21 21:47:35.957150422 +0000 UTC m=+0.802262369 container cleanup 2b223dfc5f5843b5bb880054b7d94d843bcef3fa06a259480e4936790e1dab05 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, version=8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.component=centos-stream-container) 2024-03-21T21:47:36.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:35 smithi028 bash[106708]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2 2024-03-21T21:47:36.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:36 smithi028 ceph-mon[85804]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:36.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:36 smithi028 ceph-mon[85804]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:47:36.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:36 smithi028 ceph-mon[85804]: osdmap e124: 8 total, 7 up, 8 in 2024-03-21T21:47:36.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:36 smithi028 ceph-mon[88264]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:36.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:36 smithi028 ceph-mon[88264]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:47:36.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:36 smithi028 ceph-mon[88264]: osdmap e124: 8 total, 7 up, 8 in 2024-03-21T21:47:36.698 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:36 smithi028 podman[106725]: 2024-03-21 21:47:36.283228097 +0000 UTC m=+0.664233869 container remove 2b223dfc5f5843b5bb880054b7d94d843bcef3fa06a259480e4936790e1dab05 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, version=8, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, release=754, name=centos-stream, io.openshift.expose-services=) 2024-03-21T21:47:36.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:36 smithi066 ceph-mon[68082]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:36.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:36 smithi066 ceph-mon[68082]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:47:36.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:36 smithi066 ceph-mon[68082]: osdmap e124: 8 total, 7 up, 8 in 2024-03-21T21:47:37.038 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:36 smithi028 podman[106817]: 2024-03-21 21:47:36.620871758 +0000 UTC m=+0.017379977 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:47:37.038 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:36 smithi028 podman[106817]: 2024-03-21 21:47:36.734818265 +0000 UTC m=+0.131326522 container create a98d5e3fcd366f4b3d8991272bb34879a7e85178c46b2558ce8d6babee31665a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS) 2024-03-21T21:47:37.038 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:36 smithi028 podman[106817]: 2024-03-21 21:47:36.961800915 +0000 UTC m=+0.358309129 container init a98d5e3fcd366f4b3d8991272bb34879a7e85178c46b2558ce8d6babee31665a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:47:37.038 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:36 smithi028 podman[106817]: 2024-03-21 21:47:36.964976503 +0000 UTC m=+0.361484715 container start a98d5e3fcd366f4b3d8991272bb34879a7e85178c46b2558ce8d6babee31665a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-21T21:47:37.039 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:37 smithi028 podman[106817]: 2024-03-21 21:47:37.036617721 +0000 UTC m=+0.433125935 container attach a98d5e3fcd366f4b3d8991272bb34879a7e85178c46b2558ce8d6babee31665a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-21T21:47:37.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:37 smithi028 ceph-mon[85804]: osdmap e125: 8 total, 7 up, 8 in 2024-03-21T21:47:37.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:37 smithi028 ceph-mon[88264]: osdmap e125: 8 total, 7 up, 8 in 2024-03-21T21:47:37.448 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:37 smithi028 podman[106817]: 2024-03-21 21:47:37.265167956 +0000 UTC m=+0.661676168 container died a98d5e3fcd366f4b3d8991272bb34879a7e85178c46b2558ce8d6babee31665a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-21T21:47:37.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:37 smithi066 ceph-mon[68082]: osdmap e125: 8 total, 7 up, 8 in 2024-03-21T21:47:38.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:37 smithi028 podman[106843]: 2024-03-21 21:47:37.857033792 +0000 UTC m=+0.585582938 container remove a98d5e3fcd366f4b3d8991272bb34879a7e85178c46b2558ce8d6babee31665a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-03-21T21:47:38.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:37 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-21T21:47:38.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:37 smithi028 systemd[1]: Stopped Ceph osd.2 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:47:38.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:37 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.2.service: Consumed 12.555s CPU time. 2024-03-21T21:47:38.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 systemd[1]: Starting Ceph osd.2 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:47:38.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:38 smithi028 ceph-mon[85804]: pgmap v115: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:47:38.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:38 smithi028 ceph-mon[88264]: pgmap v115: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:47:38.570 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 podman[106946]: 2024-03-21 21:47:38.299607626 +0000 UTC m=+0.113266733 container create d431e2fb4303eb44da98eca294223516dbdba9082ad04396d6fe50bdd0ade1ec (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T21:47:38.570 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 podman[106946]: 2024-03-21 21:47:38.204343616 +0000 UTC m=+0.018002732 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:47:38.570 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 podman[106946]: 2024-03-21 21:47:38.493994971 +0000 UTC m=+0.307654079 container init d431e2fb4303eb44da98eca294223516dbdba9082ad04396d6fe50bdd0ade1ec (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, GIT_BRANCH=HEAD) 2024-03-21T21:47:38.570 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 podman[106946]: 2024-03-21 21:47:38.497088574 +0000 UTC m=+0.310747693 container start d431e2fb4303eb44da98eca294223516dbdba9082ad04396d6fe50bdd0ade1ec (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-21T21:47:38.570 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 podman[106946]: 2024-03-21 21:47:38.568721541 +0000 UTC m=+0.382380659 container attach d431e2fb4303eb44da98eca294223516dbdba9082ad04396d6fe50bdd0ade1ec (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-21T21:47:38.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:38 smithi066 ceph-mon[68082]: pgmap v115: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T21:47:39.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate[106960]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T21:47:39.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 bash[106946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 bash[106946]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 bash[106946]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 bash[106946]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 bash[106946]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 bash[106946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 bash[106946]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate[106960]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate[106960]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate[106960]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate[106960]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate[106960]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate[106960]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-21T21:47:39.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:38 smithi028 podman[106946]: 2024-03-21 21:47:38.968956494 +0000 UTC m=+0.782615601 container died d431e2fb4303eb44da98eca294223516dbdba9082ad04396d6fe50bdd0ade1ec (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-21T21:47:39.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:39 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:39.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:39 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:39.651 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:39 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:39.856 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:39 smithi028 podman[107091]: 2024-03-21 21:47:39.54809101 +0000 UTC m=+0.573887279 container remove d431e2fb4303eb44da98eca294223516dbdba9082ad04396d6fe50bdd0ade1ec (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-03-21T21:47:39.856 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:39 smithi028 podman[107155]: 2024-03-21 21:47:39.757362608 +0000 UTC m=+0.112621427 container create 6ff4f4ed72aa03a9f58e4917ced558ce01016c2f23feb2ff9828d52ed19027a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:47:39.856 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:39 smithi028 podman[107155]: 2024-03-21 21:47:39.662316471 +0000 UTC m=+0.017575256 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:47:40.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:39 smithi028 podman[107155]: 2024-03-21 21:47:39.951760039 +0000 UTC m=+0.307018826 container init 6ff4f4ed72aa03a9f58e4917ced558ce01016c2f23feb2ff9828d52ed19027a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-21T21:47:40.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:39 smithi028 podman[107155]: 2024-03-21 21:47:39.95497903 +0000 UTC m=+0.310237812 container start 6ff4f4ed72aa03a9f58e4917ced558ce01016c2f23feb2ff9828d52ed19027a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T21:47:40.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:39 smithi028 ceph-osd[107173]: -- 172.21.15.28:0/1979208081 <== mon.1 v2:172.21.15.28:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cb5fe59680 con 0x55cb5fe22c00 2024-03-21T21:47:40.197 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:40 smithi028 bash[107155]: 6ff4f4ed72aa03a9f58e4917ced558ce01016c2f23feb2ff9828d52ed19027a5 2024-03-21T21:47:40.198 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:40 smithi028 systemd[1]: Started Ceph osd.2 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:47:40.697 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[107169]: 2024-03-21T21:47:40.500+0000 7f0ad26cf7c0 -1 Falling back to public interface 2024-03-21T21:47:40.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[85804]: pgmap v116: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:40.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[85804]: Health check failed: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded (PG_DEGRADED) 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[88264]: pgmap v116: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[88264]: Health check failed: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded (PG_DEGRADED) 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:40.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:40.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:40 smithi066 ceph-mon[68082]: pgmap v116: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:40.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:40 smithi066 ceph-mon[68082]: Health check failed: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded (PG_DEGRADED) 2024-03-21T21:47:40.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:40 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:40.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:40 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:40.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:40 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:40.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:40 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:41.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:47:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:47:40] "GET /metrics HTTP/1.1" 200 38167 "" "Prometheus/2.43.0" 2024-03-21T21:47:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:42 smithi028 ceph-mon[85804]: pgmap v117: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:42 smithi028 ceph-mon[88264]: pgmap v117: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:42.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:42 smithi066 ceph-mon[68082]: pgmap v117: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:43.283 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[107169]: 2024-03-21T21:47:43.224+0000 7f0ad26cf7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-21T21:47:43.284 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[107169]: 2024-03-21T21:47:43.250+0000 7f0ad26cf7c0 -1 osd.2 123 log_to_monitors true 2024-03-21T21:47:43.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:43 smithi028 ceph-mon[85804]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T21:47:43.566 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:43 smithi028 ceph-mon[88264]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T21:47:43.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:43 smithi066 ceph-mon[68082]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T21:47:43.947 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:47:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[107169]: 2024-03-21T21:47:43.564+0000 7f0abfb1f700 -1 osd.2 123 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:47:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[85804]: pgmap v118: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[85804]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T21:47:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[85804]: osdmap e126: 8 total, 7 up, 8 in 2024-03-21T21:47:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[85804]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:47:44.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[85804]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' 2024-03-21T21:47:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[88264]: pgmap v118: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[88264]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T21:47:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[88264]: osdmap e126: 8 total, 7 up, 8 in 2024-03-21T21:47:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[88264]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:47:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:44 smithi028 ceph-mon[88264]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' 2024-03-21T21:47:44.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:44 smithi066 ceph-mon[68082]: pgmap v118: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:44.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:44 smithi066 ceph-mon[68082]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T21:47:44.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:44 smithi066 ceph-mon[68082]: osdmap e126: 8 total, 7 up, 8 in 2024-03-21T21:47:44.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:44 smithi066 ceph-mon[68082]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:47:44.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:44 smithi066 ceph-mon[68082]: from='osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414]' entity='osd.2' 2024-03-21T21:47:45.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[85804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:47:46.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[85804]: osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414] boot 2024-03-21T21:47:46.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[85804]: osdmap e127: 8 total, 8 up, 8 in 2024-03-21T21:47:46.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:47:46.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:46.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:46.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[88264]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:47:46.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[88264]: osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414] boot 2024-03-21T21:47:46.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[88264]: osdmap e127: 8 total, 8 up, 8 in 2024-03-21T21:47:46.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:47:46.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:46.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:45 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:46.166 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:45 smithi066 ceph-mon[68082]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:47:46.166 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:45 smithi066 ceph-mon[68082]: osd.2 [v2:172.21.15.28:6818/97589414,v1:172.21.15.28:6819/97589414] boot 2024-03-21T21:47:46.167 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:45 smithi066 ceph-mon[68082]: osdmap e127: 8 total, 8 up, 8 in 2024-03-21T21:47:46.167 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:45 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T21:47:46.167 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:45 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:46.167 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:45 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:46.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:46 smithi028 ceph-mon[88264]: pgmap v121: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:46.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:46 smithi028 ceph-mon[88264]: osdmap e128: 8 total, 8 up, 8 in 2024-03-21T21:47:46.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:46.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:46.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:46 smithi028 ceph-mon[85804]: pgmap v121: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:46.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:46 smithi028 ceph-mon[85804]: osdmap e128: 8 total, 8 up, 8 in 2024-03-21T21:47:46.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:46.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:47.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:46 smithi066 ceph-mon[68082]: pgmap v121: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-03-21T21:47:47.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:46 smithi066 ceph-mon[68082]: osdmap e128: 8 total, 8 up, 8 in 2024-03-21T21:47:47.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:47.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:47.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:47 smithi028 ceph-mon[88264]: pgmap v123: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/627 objects degraded (9.410%) 2024-03-21T21:47:47.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:47.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:47.868 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:47 smithi028 ceph-mon[85804]: pgmap v123: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/627 objects degraded (9.410%) 2024-03-21T21:47:47.868 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:47.868 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:48.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:47 smithi066 ceph-mon[68082]: pgmap v123: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/627 objects degraded (9.410%) 2024-03-21T21:47:48.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:48.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:47:49.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:48 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:49.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:48 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:49.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:48 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[85804]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[85804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded) 2024-03-21T21:47:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[85804]: Cluster is now healthy 2024-03-21T21:47:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[88264]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[88264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded) 2024-03-21T21:47:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[88264]: Cluster is now healthy 2024-03-21T21:47:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:49 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:50.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:49 smithi066 ceph-mon[68082]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:47:50.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:49 smithi066 ceph-mon[68082]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded) 2024-03-21T21:47:50.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:49 smithi066 ceph-mon[68082]: Cluster is now healthy 2024-03-21T21:47:50.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:50.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:50.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:50.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:50.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:49 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:51.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:47:50] "GET /metrics HTTP/1.1" 200 38033 "" "Prometheus/2.43.0" 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: Upgrade: Finalizing container_image settings 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:47:51.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:47:51.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: Upgrade: Finalizing container_image settings 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:47:51.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:47:51.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: Upgrade: Complete! 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:51.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: Upgrade: Complete! 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:51.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:50 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:51.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:51.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:47:51.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: Upgrade: Finalizing container_image settings 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:47:51.252 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.253 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: Upgrade: Complete! 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:47:51.254 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:47:51.255 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:50 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:52.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:52 smithi028 ceph-mon[88264]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-21T21:47:52.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:52.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:52 smithi028 ceph-mon[85804]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-21T21:47:52.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:52.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:52 smithi066 ceph-mon[68082]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-21T21:47:52.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:54.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:54 smithi028 ceph-mon[88264]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:54.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:54 smithi028 ceph-mon[85804]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:54.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:54 smithi066 ceph-mon[68082]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:47:56.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:56 smithi028 ceph-mon[88264]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:47:56.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:56 smithi028 ceph-mon[85804]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:47:56.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:56 smithi066 ceph-mon[68082]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:47:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:57 smithi028 ceph-mon[88264]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-21T21:47:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:57 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:58.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:57 smithi028 ceph-mon[85804]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-21T21:47:58.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:57 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:58.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:57 smithi066 ceph-mon[68082]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-21T21:47:58.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:57 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:47:59.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:58 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:59.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:58 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:47:59.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:58 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:00.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:47:59 smithi028 ceph-mon[88264]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:00.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:47:59 smithi028 ceph-mon[85804]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:00.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:47:59 smithi066 ceph-mon[68082]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:00.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:00 smithi028 ceph-mon[85804]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:00.909 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:48:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:48:00] "GET /metrics HTTP/1.1" 200 38033 "" "Prometheus/2.43.0" 2024-03-21T21:48:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:00 smithi028 ceph-mon[88264]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:01.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:00 smithi066 ceph-mon[68082]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:01.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:48:02.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:01 smithi028 ceph-mon[88264]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:02.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:02.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:02.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:01 smithi028 ceph-mon[85804]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:02.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:02.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:02.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:01 smithi066 ceph-mon[68082]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:03.398 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:48:03.398 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (6m) 18s ago 28m 27.4M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:48:03.398 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (5m) 3m ago 27m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:48:03.398 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (5m) 18s ago 27m 58.0M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:48:03.398 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (7m) 3m ago 32m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:48:03.398 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (17m) 18s ago 34m 508M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:48:03.398 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (3m) 18s ago 34m 49.4M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:48:03.398 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (4m) 3m ago 33m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (3m) 18s ago 33m 40.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (6m) 18s ago 28m 15.2M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (6m) 3m ago 28m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (110s) 18s ago 32m 44.3M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (75s) 18s ago 32m 43.5M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (23s) 18s ago 31m 12.5M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (31m) 18s ago 31m 56.4M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (30m) 3m ago 30m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (30m) 3m ago 30m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (30m) 3m ago 30m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (29m) 3m ago 29m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (6m) 3m ago 29m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (27m) 18s ago 27m 78.8M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:48:03.399 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (27m) 3m ago 27m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:48:03.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-21T21:48:04.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:03 smithi028 ceph-mon[85804]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:04.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:03 smithi028 ceph-mon[85804]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:04.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:03 smithi028 ceph-mon[88264]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:04.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:03 smithi028 ceph-mon[88264]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:04.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:03 smithi066 ceph-mon[68082]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:04.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:03 smithi066 ceph-mon[68082]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:05.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:05 smithi028 ceph-mon[88264]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:05.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:05 smithi028 ceph-mon[85804]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:06.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:05 smithi066 ceph-mon[68082]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:06.318 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:48:07.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-03-21T21:48:07.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:06 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/3329649085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:07.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:06 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/3329649085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:07.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:06 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/3329649085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:08.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:07 smithi028 ceph-mon[88264]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:08.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:07 smithi028 ceph-mon[85804]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:08.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:07 smithi066 ceph-mon[68082]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:09.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:08 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:09.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:08 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:09.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:08 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:10.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:09 smithi028 ceph-mon[85804]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:10.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:09 smithi028 ceph-mon[85804]: from='client.54411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:10.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:09 smithi028 ceph-mon[88264]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:10.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:09 smithi028 ceph-mon[88264]: from='client.54411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:09 smithi066 ceph-mon[68082]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:10.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:09 smithi066 ceph-mon[68082]: from='client.54411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:11.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:48:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:48:10] "GET /metrics HTTP/1.1" 200 38172 "" "Prometheus/2.43.0" 2024-03-21T21:48:11.436 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:48:11.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade status' 2024-03-21T21:48:12.010 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:11 smithi028 ceph-mon[88264]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:12.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:11 smithi028 ceph-mon[85804]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:12.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:11 smithi066 ceph-mon[68082]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:14.002 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:48:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": null, 2024-03-21T21:48:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": false, 2024-03-21T21:48:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "", 2024-03-21T21:48:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:48:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": null, 2024-03-21T21:48:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-03-21T21:48:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:48:14.003 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:48:14.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:13 smithi028 ceph-mon[88264]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:14.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:13 smithi028 ceph-mon[85804]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:14.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:13 smithi066 ceph-mon[68082]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:14.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:48:15.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:14 smithi028 ceph-mon[85804]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:15.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:14 smithi028 ceph-mon[88264]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:15.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:14 smithi066 ceph-mon[68082]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:16.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:15 smithi028 ceph-mon[88264]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:16.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:15 smithi028 ceph-mon[85804]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:15 smithi066 ceph-mon[68082]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:16.714 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-03-21T21:48:17.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:16 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2179539372' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:48:17.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:17.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:16 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2179539372' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:48:17.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:17.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:16 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2179539372' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:48:17.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:17.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-03-21T21:48:17.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:17 smithi028 ceph-mon[85804]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:17.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:17 smithi028 ceph-mon[88264]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:18.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:17 smithi066 ceph-mon[68082]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:18 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:19.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:18 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:18 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:20.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:19 smithi066 ceph-mon[68082]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:20.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:19 smithi066 ceph-mon[68082]: from='client.54429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:20.403 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:19 smithi028 ceph-mon[88264]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:20.403 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:19 smithi028 ceph-mon[88264]: from='client.54429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:20.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:19 smithi028 ceph-mon[85804]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:20.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:19 smithi028 ceph-mon[85804]: from='client.54429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:21.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:48:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:48:20] "GET /metrics HTTP/1.1" 200 38175 "" "Prometheus/2.43.0" 2024-03-21T21:48:21.519 INFO:teuthology.orchestra.run.smithi028.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:48:22.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T21:48:22.878 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[88264]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:22.878 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[88264]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:48:22.878 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:22.878 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:22.878 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:22.878 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:48:22.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:22.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[85804]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:22.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[85804]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:48:22.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:22.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:22.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:22.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:48:22.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:22 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:23.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:22 smithi066 ceph-mon[68082]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:23.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:22 smithi066 ceph-mon[68082]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:48:23.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:22 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:23.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:22 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:23.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:22 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:23.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:22 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:48:23.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:22 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:23.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:23 smithi028 ceph-mon[85804]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:48:23.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:23 smithi028 ceph-mon[88264]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:48:24.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:23 smithi066 ceph-mon[68082]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:48:24.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:24 smithi028 ceph-mon[85804]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:24.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:24 smithi028 ceph-mon[88264]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:25.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:24 smithi066 ceph-mon[68082]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:25.510 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:48:26.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:26.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:48:26.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:48:26.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mgr 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mon 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all crash 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: Upgrade: osd.3 is safe to restart 2024-03-21T21:48:26.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[85804]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mgr 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mon 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:26.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all crash 2024-03-21T21:48:26.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T21:48:26.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T21:48:26.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: Upgrade: osd.3 is safe to restart 2024-03-21T21:48:26.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:26 smithi028 ceph-mon[88264]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (6m) 41s ago 28m 27.4M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (6m) 4m ago 28m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (5m) 41s ago 27m 58.0M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (7m) 4m ago 33m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (17m) 41s ago 34m 508M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (4m) 41s ago 35m 49.4M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (5m) 4m ago 33m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (3m) 41s ago 33m 40.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (7m) 41s ago 29m 15.2M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:48:26.207 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (7m) 4m ago 29m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (2m) 41s ago 32m 44.3M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (97s) 41s ago 32m 43.5M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (46s) 41s ago 32m 12.5M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (31m) 41s ago 31m 56.4M 2993M 17.2.0 e1d6a67b021e fd568ef9e2b4 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (31m) 4m ago 31m 55.3M 3248M 17.2.0 e1d6a67b021e ec043d17ed0e 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (30m) 4m ago 30m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (30m) 4m ago 30m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (30m) 4m ago 30m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (6m) 4m ago 29m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (27m) 41s ago 27m 78.8M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:48:26.208 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (27m) 4m ago 27m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mgr 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:26.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mon 2024-03-21T21:48:26.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:26.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all crash 2024-03-21T21:48:26.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:26.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T21:48:26.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T21:48:26.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: Upgrade: osd.3 is safe to restart 2024-03-21T21:48:26.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:26 smithi066 ceph-mon[68082]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:48:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:48:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:48:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:48:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:48:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-21T21:48:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 8 2024-03-21T21:48:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:48:26.623 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:48:26.976 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:48:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:48:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:48:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T21:48:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:48:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "3/8 daemons upgraded", 2024-03-21T21:48:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T21:48:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:48:26.977 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:48:27.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[85804]: from='client.54441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:27.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[85804]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:27.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/3845264858' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:27.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:27.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T21:48:27.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:27.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[88264]: from='client.54441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:27.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[88264]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:27.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/3845264858' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:27.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:27.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T21:48:27.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:27 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:27.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:27 smithi066 ceph-mon[68082]: from='client.54441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:27.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:27 smithi066 ceph-mon[68082]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:27.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:27 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/3845264858' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:27.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:27.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T21:48:27.502 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:28.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:28 smithi028 ceph-mon[88264]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:28.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:28 smithi028 ceph-mon[88264]: Upgrade: Updating osd.3 2024-03-21T21:48:28.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:28 smithi028 ceph-mon[88264]: Deploying daemon osd.3 on smithi028 2024-03-21T21:48:28.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:28 smithi028 ceph-mon[88264]: from='client.64396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:28.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:28 smithi028 ceph-mon[85804]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:28.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:28 smithi028 ceph-mon[85804]: Upgrade: Updating osd.3 2024-03-21T21:48:28.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:28 smithi028 ceph-mon[85804]: Deploying daemon osd.3 on smithi028 2024-03-21T21:48:28.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:28 smithi028 ceph-mon[85804]: from='client.64396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:28.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:28 smithi066 ceph-mon[68082]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:28.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:28 smithi066 ceph-mon[68082]: Upgrade: Updating osd.3 2024-03-21T21:48:28.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:28 smithi066 ceph-mon[68082]: Deploying daemon osd.3 on smithi028 2024-03-21T21:48:28.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:28 smithi066 ceph-mon[68082]: from='client.64396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:29.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:29 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:29.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:29 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:29.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:29 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:30.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:30 smithi028 ceph-mon[85804]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:30.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:30 smithi028 ceph-mon[88264]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:30.388 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:30 smithi028 systemd[1]: Stopping Ceph osd.3 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:48:30.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:30 smithi066 ceph-mon[68082]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:48:30.697 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[41298]: 2024-03-21T21:48:30.385+0000 7f6d3afe4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:48:30.697 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[41298]: 2024-03-21T21:48:30.385+0000 7f6d3afe4700 -1 osd.3 128 *** Got signal Terminated *** 2024-03-21T21:48:30.697 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[41298]: 2024-03-21T21:48:30.385+0000 7f6d3afe4700 -1 osd.3 128 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:48:31.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:31 smithi028 ceph-mon[85804]: osd.3 marked itself down and dead 2024-03-21T21:48:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:31 smithi028 ceph-mon[88264]: osd.3 marked itself down and dead 2024-03-21T21:48:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:48:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:48:30] "GET /metrics HTTP/1.1" 200 38175 "" "Prometheus/2.43.0" 2024-03-21T21:48:31.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:31 smithi066 ceph-mon[68082]: osd.3 marked itself down and dead 2024-03-21T21:48:31.697 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:31 smithi028 podman[113095]: 2024-03-21 21:48:31.438476716 +0000 UTC m=+1.167460936 container stop fd568ef9e2b4d984b967971ac060515e5b4fb8d10d61a573d5292ef2f43f8537 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, ceph=True, version=8, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vcs-type=git, RELEASE=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , name=centos-stream, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0) 2024-03-21T21:48:31.697 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:31 smithi028 podman[113095]: 2024-03-21 21:48:31.464313177 +0000 UTC m=+1.193297390 container died fd568ef9e2b4d984b967971ac060515e5b4fb8d10d61a573d5292ef2f43f8537 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, distribution-scope=public, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, ceph=True, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., vcs-type=git, release=754, version=8, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-21T21:48:32.085 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:31 smithi028 podman[113095]: 2024-03-21 21:48:31.794356309 +0000 UTC m=+1.523340520 container cleanup fd568ef9e2b4d984b967971ac060515e5b4fb8d10d61a573d5292ef2f43f8537 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, architecture=x86_64, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_BRANCH=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc.) 2024-03-21T21:48:32.085 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:31 smithi028 bash[113095]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3 2024-03-21T21:48:32.085 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:31 smithi028 podman[113113]: 2024-03-21 21:48:31.945538155 +0000 UTC m=+0.500532143 container remove fd568ef9e2b4d984b967971ac060515e5b4fb8d10d61a573d5292ef2f43f8537 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, distribution-scope=public, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., name=centos-stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-21T21:48:32.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[88264]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:32.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[88264]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:48:32.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[88264]: osdmap e129: 8 total, 7 up, 8 in 2024-03-21T21:48:32.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:32.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:32.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[85804]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:32.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[85804]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:48:32.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[85804]: osdmap e129: 8 total, 7 up, 8 in 2024-03-21T21:48:32.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:32.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:32.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:32 smithi066 ceph-mon[68082]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:48:32.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:32 smithi066 ceph-mon[68082]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:48:32.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:32 smithi066 ceph-mon[68082]: osdmap e129: 8 total, 7 up, 8 in 2024-03-21T21:48:32.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:32.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:32.667 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:32 smithi028 podman[113207]: 2024-03-21 21:48:32.363192297 +0000 UTC m=+0.109404777 container create 3a9b1e7fd09d0b8b0a3120cb0b324b1bcec667433a41fecb86ba3f7b30e6d52b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-21T21:48:32.667 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:32 smithi028 podman[113207]: 2024-03-21 21:48:32.271804675 +0000 UTC m=+0.018017179 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:48:32.667 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:32 smithi028 podman[113207]: 2024-03-21 21:48:32.599536883 +0000 UTC m=+0.345749373 container init 3a9b1e7fd09d0b8b0a3120cb0b324b1bcec667433a41fecb86ba3f7b30e6d52b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-21T21:48:32.667 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:32 smithi028 podman[113207]: 2024-03-21 21:48:32.60263897 +0000 UTC m=+0.348851449 container start 3a9b1e7fd09d0b8b0a3120cb0b324b1bcec667433a41fecb86ba3f7b30e6d52b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:48:32.667 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:32 smithi028 podman[113207]: 2024-03-21 21:48:32.665999088 +0000 UTC m=+0.412211575 container attach 3a9b1e7fd09d0b8b0a3120cb0b324b1bcec667433a41fecb86ba3f7b30e6d52b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T21:48:32.947 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:32 smithi028 podman[113207]: 2024-03-21 21:48:32.901568588 +0000 UTC m=+0.647781067 container died 3a9b1e7fd09d0b8b0a3120cb0b324b1bcec667433a41fecb86ba3f7b30e6d52b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-03-21T21:48:33.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:33 smithi028 ceph-mon[88264]: osdmap e130: 8 total, 7 up, 8 in 2024-03-21T21:48:33.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:33 smithi028 ceph-mon[85804]: osdmap e130: 8 total, 7 up, 8 in 2024-03-21T21:48:33.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:33 smithi066 ceph-mon[68082]: osdmap e130: 8 total, 7 up, 8 in 2024-03-21T21:48:33.907 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:33 smithi028 podman[113233]: 2024-03-21 21:48:33.644752594 +0000 UTC m=+0.738423464 container remove 3a9b1e7fd09d0b8b0a3120cb0b324b1bcec667433a41fecb86ba3f7b30e6d52b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-21T21:48:33.907 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:33 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-21T21:48:33.907 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:33 smithi028 systemd[1]: Stopped Ceph osd.3 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:48:33.908 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:33 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.3.service: Consumed 17.339s CPU time. 2024-03-21T21:48:33.908 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:33 smithi028 systemd[1]: Starting Ceph osd.3 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:48:34.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-mon[88264]: pgmap v148: 161 pgs: 12 active+undersized+degraded, 13 stale+active+clean, 20 active+undersized, 116 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-03-21T21:48:34.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-mon[88264]: Health check failed: Degraded data redundancy: 38/627 objects degraded (6.061%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T21:48:34.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-mon[85804]: pgmap v148: 161 pgs: 12 active+undersized+degraded, 13 stale+active+clean, 20 active+undersized, 116 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-03-21T21:48:34.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-mon[85804]: Health check failed: Degraded data redundancy: 38/627 objects degraded (6.061%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T21:48:34.197 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 podman[113334]: 2024-03-21 21:48:34.01245933 +0000 UTC m=+0.018671733 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:48:34.198 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 podman[113334]: 2024-03-21 21:48:34.1127056 +0000 UTC m=+0.118917986 container create cced4aec4c511559b207429c488dc8c60afd18b56e03996448489de78c7f1322 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True) 2024-03-21T21:48:34.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:34 smithi066 ceph-mon[68082]: pgmap v148: 161 pgs: 12 active+undersized+degraded, 13 stale+active+clean, 20 active+undersized, 116 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-03-21T21:48:34.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:34 smithi066 ceph-mon[68082]: Health check failed: Degraded data redundancy: 38/627 objects degraded (6.061%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T21:48:34.697 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 podman[113334]: 2024-03-21 21:48:34.332231646 +0000 UTC m=+0.338444043 container init cced4aec4c511559b207429c488dc8c60afd18b56e03996448489de78c7f1322 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:48:34.697 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 podman[113334]: 2024-03-21 21:48:34.33660459 +0000 UTC m=+0.342816978 container start cced4aec4c511559b207429c488dc8c60afd18b56e03996448489de78c7f1322 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-21T21:48:34.697 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 podman[113334]: 2024-03-21 21:48:34.40714624 +0000 UTC m=+0.413358631 container attach cced4aec4c511559b207429c488dc8c60afd18b56e03996448489de78c7f1322 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate[113356]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate[113356]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 bash[113334]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 bash[113334]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 bash[113334]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 bash[113334]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 bash[113334]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 bash[113334]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 bash[113334]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate[113356]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-21T21:48:35.179 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate[113356]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-21T21:48:35.180 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate[113356]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-21T21:48:35.180 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate[113356]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-21T21:48:35.180 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate[113356]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-21T21:48:35.180 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:34 smithi028 podman[113334]: 2024-03-21 21:48:34.813034653 +0000 UTC m=+0.819247050 container died cced4aec4c511559b207429c488dc8c60afd18b56e03996448489de78c7f1322 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-03-21T21:48:35.781 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:35 smithi028 podman[113495]: 2024-03-21 21:48:35.469388323 +0000 UTC m=+0.651660769 container remove cced4aec4c511559b207429c488dc8c60afd18b56e03996448489de78c7f1322 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:48:35.781 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:35 smithi028 podman[113559]: 2024-03-21 21:48:35.58888584 +0000 UTC m=+0.017978857 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:48:35.781 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:35 smithi028 podman[113559]: 2024-03-21 21:48:35.686747396 +0000 UTC m=+0.115840405 container create d5d9fb92993d4190b94978299f8f9d52197ae9fcee8c1d4f383bda2f0d305063 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:48:36.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:36 smithi028 ceph-mon[85804]: pgmap v149: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:36.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:36.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:36.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:36.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:36 smithi028 ceph-mon[88264]: pgmap v149: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:36.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:36.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:36.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:36.198 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:35 smithi028 podman[113559]: 2024-03-21 21:48:35.88983322 +0000 UTC m=+0.318926236 container init d5d9fb92993d4190b94978299f8f9d52197ae9fcee8c1d4f383bda2f0d305063 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T21:48:36.198 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:35 smithi028 podman[113559]: 2024-03-21 21:48:35.892906519 +0000 UTC m=+0.321999528 container start d5d9fb92993d4190b94978299f8f9d52197ae9fcee8c1d4f383bda2f0d305063 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T21:48:36.198 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:35 smithi028 ceph-osd[113578]: -- 172.21.15.28:0/848714250 <== mon.1 v2:172.21.15.28:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e339819680 con 0x55e3397e2c00 2024-03-21T21:48:36.198 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:35 smithi028 bash[113559]: d5d9fb92993d4190b94978299f8f9d52197ae9fcee8c1d4f383bda2f0d305063 2024-03-21T21:48:36.198 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:35 smithi028 systemd[1]: Started Ceph osd.3 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:48:36.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:36 smithi066 ceph-mon[68082]: pgmap v149: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:36.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:36.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:36.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:36.697 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:36 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[113574]: 2024-03-21T21:48:36.430+0000 7f1cdfac07c0 -1 Falling back to public interface 2024-03-21T21:48:38.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:38 smithi028 ceph-mon[88264]: pgmap v150: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:38.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:38 smithi028 ceph-mon[85804]: pgmap v150: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:39.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:38 smithi066 ceph-mon[68082]: pgmap v150: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:39.447 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[113574]: 2024-03-21T21:48:39.140+0000 7f1cdfac07c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-21T21:48:39.447 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[113574]: 2024-03-21T21:48:39.169+0000 7f1cdfac07c0 -1 osd.3 128 log_to_monitors true 2024-03-21T21:48:39.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:39 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:39.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:39 smithi028 ceph-mon[88264]: from='osd.3 [v2:172.21.15.28:6826/120151747,v1:172.21.15.28:6827/120151747]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T21:48:39.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:39 smithi028 ceph-mon[88264]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T21:48:39.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:39 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:39.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:39 smithi028 ceph-mon[85804]: from='osd.3 [v2:172.21.15.28:6826/120151747,v1:172.21.15.28:6827/120151747]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T21:48:39.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:39 smithi028 ceph-mon[85804]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T21:48:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:39 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:39 smithi066 ceph-mon[68082]: from='osd.3 [v2:172.21.15.28:6826/120151747,v1:172.21.15.28:6827/120151747]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T21:48:40.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:39 smithi066 ceph-mon[68082]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T21:48:40.197 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:48:39 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[113574]: 2024-03-21T21:48:39.850+0000 7f1cccf10700 -1 osd.3 128 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:48:40.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[88264]: pgmap v151: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:40.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[88264]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T21:48:40.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[88264]: osdmap e131: 8 total, 7 up, 8 in 2024-03-21T21:48:40.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[88264]: from='osd.3 [v2:172.21.15.28:6826/120151747,v1:172.21.15.28:6827/120151747]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:48:40.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[88264]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:48:40.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[88264]: from='osd.3 ' entity='osd.3' 2024-03-21T21:48:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[85804]: pgmap v151: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[85804]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T21:48:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[85804]: osdmap e131: 8 total, 7 up, 8 in 2024-03-21T21:48:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[85804]: from='osd.3 [v2:172.21.15.28:6826/120151747,v1:172.21.15.28:6827/120151747]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:48:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[85804]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:48:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-mon[85804]: from='osd.3 ' entity='osd.3' 2024-03-21T21:48:41.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:40 smithi066 ceph-mon[68082]: pgmap v151: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:40 smithi066 ceph-mon[68082]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T21:48:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:40 smithi066 ceph-mon[68082]: osdmap e131: 8 total, 7 up, 8 in 2024-03-21T21:48:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:40 smithi066 ceph-mon[68082]: from='osd.3 [v2:172.21.15.28:6826/120151747,v1:172.21.15.28:6827/120151747]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:48:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:40 smithi066 ceph-mon[68082]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-21T21:48:41.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:40 smithi066 ceph-mon[68082]: from='osd.3 ' entity='osd.3' 2024-03-21T21:48:41.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:48:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:48:40] "GET /metrics HTTP/1.1" 200 38173 "" "Prometheus/2.43.0" 2024-03-21T21:48:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:41 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:42.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:41 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:42.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:41 smithi066 ceph-mon[68082]: pgmap v153: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:42.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:41 smithi066 ceph-mon[68082]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:48:42.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:41 smithi066 ceph-mon[68082]: osd.3 [v2:172.21.15.28:6826/120151747,v1:172.21.15.28:6827/120151747] boot 2024-03-21T21:48:42.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:41 smithi066 ceph-mon[68082]: osdmap e132: 8 total, 8 up, 8 in 2024-03-21T21:48:42.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:41 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:48:42.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:41 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 24 pgs degraded (PG_DEGRADED) 2024-03-21T21:48:42.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:42.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:42.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[88264]: pgmap v153: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:42.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[88264]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:48:42.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[88264]: osd.3 [v2:172.21.15.28:6826/120151747,v1:172.21.15.28:6827/120151747] boot 2024-03-21T21:48:42.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[88264]: osdmap e132: 8 total, 8 up, 8 in 2024-03-21T21:48:42.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:48:42.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 24 pgs degraded (PG_DEGRADED) 2024-03-21T21:48:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[85804]: pgmap v153: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T21:48:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[85804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:48:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[85804]: osd.3 [v2:172.21.15.28:6826/120151747,v1:172.21.15.28:6827/120151747] boot 2024-03-21T21:48:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[85804]: osdmap e132: 8 total, 8 up, 8 in 2024-03-21T21:48:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T21:48:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:41 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 24 pgs degraded (PG_DEGRADED) 2024-03-21T21:48:43.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:42 smithi028 ceph-mon[88264]: osdmap e133: 8 total, 8 up, 8 in 2024-03-21T21:48:43.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:43.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:43.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:42 smithi028 ceph-mon[85804]: osdmap e133: 8 total, 8 up, 8 in 2024-03-21T21:48:43.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:43.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:43.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:42 smithi066 ceph-mon[68082]: osdmap e133: 8 total, 8 up, 8 in 2024-03-21T21:48:43.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:43.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:44.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:43 smithi028 ceph-mon[88264]: pgmap v156: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-21T21:48:44.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:43 smithi028 ceph-mon[85804]: pgmap v156: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-21T21:48:44.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:43 smithi066 ceph-mon[68082]: pgmap v156: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-21T21:48:45.040 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:44 smithi028 ceph-mon[88264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/627 objects degraded (9.250%), 17 pgs degraded) 2024-03-21T21:48:45.040 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:44 smithi028 ceph-mon[88264]: Cluster is now healthy 2024-03-21T21:48:45.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:44 smithi028 ceph-mon[85804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/627 objects degraded (9.250%), 17 pgs degraded) 2024-03-21T21:48:45.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:44 smithi028 ceph-mon[85804]: Cluster is now healthy 2024-03-21T21:48:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:44 smithi066 ceph-mon[68082]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/627 objects degraded (9.250%), 17 pgs degraded) 2024-03-21T21:48:45.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:44 smithi066 ceph-mon[68082]: Cluster is now healthy 2024-03-21T21:48:46.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:48:46.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:46.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:46.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:46.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:48:46.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:46.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:46.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:46.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:46.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:46.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T21:48:46.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T21:48:46.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:46 smithi066 ceph-mon[68082]: Upgrade: osd.4 is safe to restart 2024-03-21T21:48:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:48:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:46.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[85804]: Upgrade: osd.4 is safe to restart 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:48:46.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T21:48:46.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:46 smithi028 ceph-mon[88264]: Upgrade: osd.4 is safe to restart 2024-03-21T21:48:48.156 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:47 smithi066 ceph-mon[68082]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:48:48.157 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:48.157 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:48.157 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:47 smithi066 ceph-mon[68082]: Upgrade: Updating osd.4 2024-03-21T21:48:48.157 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:48.157 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T21:48:48.157 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:48.157 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:47 smithi066 ceph-mon[68082]: Deploying daemon osd.4 on smithi066 2024-03-21T21:48:48.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[85804]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:48:48.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:48.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:48.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[85804]: Upgrade: Updating osd.4 2024-03-21T21:48:48.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:48.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T21:48:48.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:48.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[85804]: Deploying daemon osd.4 on smithi066 2024-03-21T21:48:48.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[88264]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:48:48.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:48.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:48:48.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[88264]: Upgrade: Updating osd.4 2024-03-21T21:48:48.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:48.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T21:48:48.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:48:48.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:47 smithi028 ceph-mon[88264]: Deploying daemon osd.4 on smithi066 2024-03-21T21:48:49.112 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:48 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:49.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:48 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:49.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:48 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:50.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:49 smithi066 ceph-mon[68082]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:48:50.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:49 smithi028 ceph-mon[88264]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:48:50.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:49 smithi028 ceph-mon[85804]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:48:51.000 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:50 smithi066 systemd[1]: Stopping Ceph osd.4 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:48:51.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:48:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:48:50] "GET /metrics HTTP/1.1" 200 38180 "" "Prometheus/2.43.0" 2024-03-21T21:48:51.250 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[29040]: 2024-03-21T21:48:51.027+0000 7fca6444a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:48:51.250 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[29040]: 2024-03-21T21:48:51.027+0000 7fca6444a700 -1 osd.4 133 *** Got signal Terminated *** 2024-03-21T21:48:51.251 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:51 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[29040]: 2024-03-21T21:48:51.027+0000 7fca6444a700 -1 osd.4 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:48:52.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:51 smithi028 ceph-mon[88264]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2024-03-21T21:48:52.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:51 smithi028 ceph-mon[88264]: osd.4 marked itself down and dead 2024-03-21T21:48:52.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:51 smithi028 ceph-mon[85804]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2024-03-21T21:48:52.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:51 smithi028 ceph-mon[85804]: osd.4 marked itself down and dead 2024-03-21T21:48:52.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:51 smithi066 ceph-mon[68082]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2024-03-21T21:48:52.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:51 smithi066 ceph-mon[68082]: osd.4 marked itself down and dead 2024-03-21T21:48:52.594 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:52 smithi066 podman[77945]: 2024-03-21 21:48:52.301785209 +0000 UTC m=+1.387970733 container died ec043d17ed0e2517a67a28be5e5df7cb64772d6b7147a6fadbb2d8497b7ea598 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.openshift.tags=base centos centos-stream, ceph=True) 2024-03-21T21:48:52.969 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:52 smithi066 ceph-mon[68082]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:48:52.970 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:52 smithi066 ceph-mon[68082]: osdmap e134: 8 total, 7 up, 8 in 2024-03-21T21:48:52.970 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:52.970 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:52 smithi066 podman[77945]: 2024-03-21 21:48:52.700670926 +0000 UTC m=+1.786856487 container cleanup ec043d17ed0e2517a67a28be5e5df7cb64772d6b7147a6fadbb2d8497b7ea598 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4, vcs-type=git, RELEASE=HEAD, vendor=Red Hat, Inc., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, release=754, name=centos-stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , ceph=True) 2024-03-21T21:48:52.970 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:52 smithi066 bash[77945]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4 2024-03-21T21:48:53.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:52 smithi028 ceph-mon[88264]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:48:53.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:52 smithi028 ceph-mon[88264]: osdmap e134: 8 total, 7 up, 8 in 2024-03-21T21:48:53.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:53.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:52 smithi028 ceph-mon[85804]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:48:53.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:52 smithi028 ceph-mon[85804]: osdmap e134: 8 total, 7 up, 8 in 2024-03-21T21:48:53.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:53.250 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:52 smithi066 podman[77973]: 2024-03-21 21:48:52.96891481 +0000 UTC m=+0.666573478 container remove ec043d17ed0e2517a67a28be5e5df7cb64772d6b7147a6fadbb2d8497b7ea598 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , name=centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.buildah.version=1.19.8, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-21T21:48:53.536 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:53 smithi066 podman[78067]: 2024-03-21 21:48:53.186813301 +0000 UTC m=+0.018242125 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:48:53.537 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:53 smithi066 podman[78067]: 2024-03-21 21:48:53.286505777 +0000 UTC m=+0.117934579 container create b0d7d6827c237249e694c460f4daadb8f63edf76f3465a67a74264a01c1dc0b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-21T21:48:53.537 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:53 smithi066 podman[78067]: 2024-03-21 21:48:53.532847688 +0000 UTC m=+0.364276491 container init b0d7d6827c237249e694c460f4daadb8f63edf76f3465a67a74264a01c1dc0b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-21T21:48:53.537 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:53 smithi066 podman[78067]: 2024-03-21 21:48:53.536234519 +0000 UTC m=+0.367663330 container start b0d7d6827c237249e694c460f4daadb8f63edf76f3465a67a74264a01c1dc0b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS) 2024-03-21T21:48:53.841 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:53 smithi066 podman[78067]: 2024-03-21 21:48:53.639363352 +0000 UTC m=+0.470792166 container attach b0d7d6827c237249e694c460f4daadb8f63edf76f3465a67a74264a01c1dc0b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-21T21:48:54.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:53 smithi066 ceph-mon[68082]: pgmap v162: 161 pgs: 33 peering, 13 stale+active+clean, 115 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-03-21T21:48:54.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:53 smithi066 ceph-mon[68082]: osdmap e135: 8 total, 7 up, 8 in 2024-03-21T21:48:54.151 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:53 smithi066 ceph-mon[68082]: Health check failed: Reduced data availability: 6 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-21T21:48:54.152 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:53 smithi066 podman[78067]: 2024-03-21 21:48:53.84061232 +0000 UTC m=+0.672041131 container died b0d7d6827c237249e694c460f4daadb8f63edf76f3465a67a74264a01c1dc0b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-21T21:48:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:53 smithi028 ceph-mon[88264]: pgmap v162: 161 pgs: 33 peering, 13 stale+active+clean, 115 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-03-21T21:48:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:53 smithi028 ceph-mon[88264]: osdmap e135: 8 total, 7 up, 8 in 2024-03-21T21:48:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:53 smithi028 ceph-mon[88264]: Health check failed: Reduced data availability: 6 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-21T21:48:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:53 smithi028 ceph-mon[85804]: pgmap v162: 161 pgs: 33 peering, 13 stale+active+clean, 115 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-03-21T21:48:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:53 smithi028 ceph-mon[85804]: osdmap e135: 8 total, 7 up, 8 in 2024-03-21T21:48:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:53 smithi028 ceph-mon[85804]: Health check failed: Reduced data availability: 6 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-21T21:48:54.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:54 smithi066 podman[78093]: 2024-03-21 21:48:54.443006931 +0000 UTC m=+0.596194767 container remove b0d7d6827c237249e694c460f4daadb8f63edf76f3465a67a74264a01c1dc0b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-21T21:48:54.751 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:54 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-21T21:48:54.751 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:54 smithi066 systemd[1]: Stopped Ceph osd.4 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:48:54.751 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:54 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.4.service: Consumed 17.836s CPU time. 2024-03-21T21:48:54.751 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:54 smithi066 systemd[1]: Starting Ceph osd.4 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:48:55.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:54 smithi028 ceph-mon[85804]: Health check failed: Degraded data redundancy: 47/627 objects degraded (7.496%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T21:48:55.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:54 smithi028 ceph-mon[88264]: Health check failed: Degraded data redundancy: 47/627 objects degraded (7.496%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T21:48:55.198 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:54 smithi066 ceph-mon[68082]: Health check failed: Degraded data redundancy: 47/627 objects degraded (7.496%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T21:48:55.198 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:54 smithi066 podman[78201]: 2024-03-21 21:48:54.816632352 +0000 UTC m=+0.019044613 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:48:55.198 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:54 smithi066 podman[78201]: 2024-03-21 21:48:54.935883877 +0000 UTC m=+0.138296122 container create fc313a3f9436df7fd079a119b73ec8ae5d5027c87ef7ab5dc31be6ba8e5a7472 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:48:55.500 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 podman[78201]: 2024-03-21 21:48:55.197346024 +0000 UTC m=+0.399758271 container init fc313a3f9436df7fd079a119b73ec8ae5d5027c87ef7ab5dc31be6ba8e5a7472 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True) 2024-03-21T21:48:55.501 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 podman[78201]: 2024-03-21 21:48:55.20129236 +0000 UTC m=+0.403704609 container start fc313a3f9436df7fd079a119b73ec8ae5d5027c87ef7ab5dc31be6ba8e5a7472 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:48:55.501 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 podman[78201]: 2024-03-21 21:48:55.288606025 +0000 UTC m=+0.491018279 container attach fc313a3f9436df7fd079a119b73ec8ae5d5027c87ef7ab5dc31be6ba8e5a7472 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:48:55.992 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:55 smithi066 ceph-mon[68082]: pgmap v164: 161 pgs: 12 active+undersized+degraded, 33 peering, 23 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 47/627 objects degraded (7.496%) 2024-03-21T21:48:55.993 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate[78216]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-21T21:48:55.993 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate[78216]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-21T21:48:55.993 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 bash[78201]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-21T21:48:55.993 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 bash[78201]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-21T21:48:55.993 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 bash[78201]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-21T21:48:55.993 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 bash[78201]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-21T21:48:55.993 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 bash[78201]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-21T21:48:55.993 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 bash[78201]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-21T21:48:55.993 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 bash[78201]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-21T21:48:55.994 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate[78216]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-21T21:48:55.994 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate[78216]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-21T21:48:55.994 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate[78216]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-21T21:48:55.994 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate[78216]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-21T21:48:55.994 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate[78216]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-21T21:48:55.994 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:55 smithi066 podman[78201]: 2024-03-21 21:48:55.6799985 +0000 UTC m=+0.882410743 container died fc313a3f9436df7fd079a119b73ec8ae5d5027c87ef7ab5dc31be6ba8e5a7472 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-21T21:48:56.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:55 smithi028 ceph-mon[85804]: pgmap v164: 161 pgs: 12 active+undersized+degraded, 33 peering, 23 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 47/627 objects degraded (7.496%) 2024-03-21T21:48:56.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:55 smithi028 ceph-mon[88264]: pgmap v164: 161 pgs: 12 active+undersized+degraded, 33 peering, 23 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 47/627 objects degraded (7.496%) 2024-03-21T21:48:56.560 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:56 smithi066 podman[78333]: 2024-03-21 21:48:56.308945905 +0000 UTC m=+0.621343586 container remove fc313a3f9436df7fd079a119b73ec8ae5d5027c87ef7ab5dc31be6ba8e5a7472 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-21T21:48:56.560 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:56 smithi066 podman[78394]: 2024-03-21 21:48:56.43658247 +0000 UTC m=+0.018890791 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:48:56.560 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:56 smithi066 podman[78394]: 2024-03-21 21:48:56.559965898 +0000 UTC m=+0.142274201 container create ecd87bfd3243118f47e47e0b4552b185d1ff747be10eeeba0f289f33d10b6ce7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-21T21:48:56.829 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:56 smithi066 podman[78394]: 2024-03-21 21:48:56.804449549 +0000 UTC m=+0.386757862 container init ecd87bfd3243118f47e47e0b4552b185d1ff747be10eeeba0f289f33d10b6ce7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-21T21:48:56.829 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:56 smithi066 podman[78394]: 2024-03-21 21:48:56.807843672 +0000 UTC m=+0.390151976 container start ecd87bfd3243118f47e47e0b4552b185d1ff747be10eeeba0f289f33d10b6ce7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-21T21:48:57.250 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:56 smithi066 ceph-osd[78413]: -- 172.21.15.66:0/2445285945 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556def62b680 con 0x556def5f4c00 2024-03-21T21:48:57.251 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:56 smithi066 bash[78394]: ecd87bfd3243118f47e47e0b4552b185d1ff747be10eeeba0f289f33d10b6ce7 2024-03-21T21:48:57.251 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:56 smithi066 systemd[1]: Started Ceph osd.4 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:48:57.350 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:48:57.754 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:48:57 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:48:57.352+0000 7ff63c0a47c0 -1 Falling back to public interface 2024-03-21T21:48:58.031 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:48:58.031 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (6m) 17s ago 29m 27.5M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:48:58.031 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (6m) 4m ago 28m 84.2M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:48:58.032 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (6m) 17s ago 28m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:48:58.032 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (8m) 4m ago 33m 448M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:48:58.032 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (18m) 17s ago 35m 510M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:48:58.032 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (4m) 17s ago 35m 50.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:48:58.032 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (5m) 4m ago 34m 37.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:48:58.032 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (4m) 17s ago 34m 40.5M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:48:58.032 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (7m) 17s ago 29m 15.2M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:48:58.032 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (7m) 4m ago 29m 14.7M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:48:58.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (2m) 17s ago 33m 46.0M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:48:58.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (2m) 17s ago 33m 44.2M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:48:58.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (78s) 17s ago 32m 49.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:48:58.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (22s) 17s ago 32m 15.1M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:48:58.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 starting - - - 3248M 2024-03-21T21:48:58.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (31m) 4m ago 31m 52.9M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:48:58.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (31m) 4m ago 31m 52.4M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:48:58.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (30m) 4m ago 30m 53.0M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:48:58.034 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (7m) 4m ago 29m 69.1M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:48:58.034 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (28m) 17s ago 28m 79.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:48:58.034 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (28m) 4m ago 28m 76.4M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:48:58.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[85804]: pgmap v165: 161 pgs: 12 active+undersized+degraded, 33 peering, 23 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 47/627 objects degraded (7.496%) 2024-03-21T21:48:58.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:58.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:58.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:58.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[85804]: from='client.54471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[88264]: pgmap v165: 161 pgs: 12 active+undersized+degraded, 33 peering, 23 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 47/627 objects degraded (7.496%) 2024-03-21T21:48:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:58.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:57 smithi028 ceph-mon[88264]: from='client.54471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:57 smithi066 ceph-mon[68082]: pgmap v165: 161 pgs: 12 active+undersized+degraded, 33 peering, 23 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 47/627 objects degraded (7.496%) 2024-03-21T21:48:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:57 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:57 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:48:58.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:57 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:48:58.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:57 smithi066 ceph-mon[68082]: from='client.54471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 4 2024-03-21T21:48:58.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:48:58.443 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:48:58.443 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:48:58.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:48:58.443 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:48:58.443 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-21T21:48:58.443 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 9 2024-03-21T21:48:58.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:48:58.443 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:48:58.785 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:48:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:48:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:48:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T21:48:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:48:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "4/8 daemons upgraded", 2024-03-21T21:48:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T21:48:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:48:58.786 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:48:59.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:58 smithi028 ceph-mon[88264]: from='client.54477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:59.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:58 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:59.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:58 smithi028 ceph-mon[88264]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:59.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:58 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/1572561290' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:59.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:58 smithi028 ceph-mon[85804]: from='client.54477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:59.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:58 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:59.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:58 smithi028 ceph-mon[85804]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:59.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:58 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/1572561290' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:48:59.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:58 smithi066 ceph-mon[68082]: from='client.54477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:59.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:58 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:48:59.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:58 smithi066 ceph-mon[68082]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:48:59.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:58 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/1572561290' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:00.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:59 smithi028 ceph-mon[88264]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:00.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:59 smithi028 ceph-mon[88264]: pgmap v166: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-21T21:49:00.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:48:59 smithi028 ceph-mon[88264]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 10 pgs peering) 2024-03-21T21:49:00.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:59 smithi028 ceph-mon[85804]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:00.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:59 smithi028 ceph-mon[85804]: pgmap v166: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-21T21:49:00.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:48:59 smithi028 ceph-mon[85804]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 10 pgs peering) 2024-03-21T21:49:00.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:59 smithi066 ceph-mon[68082]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:00.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:59 smithi066 ceph-mon[68082]: pgmap v166: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-21T21:49:00.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:48:59 smithi066 ceph-mon[68082]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 10 pgs peering) 2024-03-21T21:49:00.251 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:49:00 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:49:00.086+0000 7ff63c0a47c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-21T21:49:00.251 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:49:00 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:49:00.120+0000 7ff63c0a47c0 -1 osd.4 133 log_to_monitors true 2024-03-21T21:49:01.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:49:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:49:00] "GET /metrics HTTP/1.1" 200 38180 "" "Prometheus/2.43.0" 2024-03-21T21:49:01.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:00 smithi028 ceph-mon[85804]: from='osd.4 [v2:172.21.15.66:6800/163594816,v1:172.21.15.66:6801/163594816]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:49:01.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:00 smithi028 ceph-mon[85804]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:49:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:00 smithi028 ceph-mon[88264]: from='osd.4 [v2:172.21.15.66:6800/163594816,v1:172.21.15.66:6801/163594816]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:49:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:00 smithi028 ceph-mon[88264]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:49:01.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:00 smithi066 ceph-mon[68082]: from='osd.4 [v2:172.21.15.66:6800/163594816,v1:172.21.15.66:6801/163594816]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:49:01.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:00 smithi066 ceph-mon[68082]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T21:49:01.250 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:49:00 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:49:00.995+0000 7ff62d4fc700 -1 osd.4 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:49:02.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: pgmap v167: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-21T21:49:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T21:49:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: osdmap e136: 8 total, 7 up, 8 in 2024-03-21T21:49:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: from='osd.4 [v2:172.21.15.66:6800/163594816,v1:172.21.15.66:6801/163594816]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 27 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:02.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: pgmap v167: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-21T21:49:02.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T21:49:02.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: osdmap e136: 8 total, 7 up, 8 in 2024-03-21T21:49:02.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: from='osd.4 [v2:172.21.15.66:6800/163594816,v1:172.21.15.66:6801/163594816]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:02.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:02.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:02.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:02.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 27 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: pgmap v167: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: osdmap e136: 8 total, 7 up, 8 in 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: from='osd.4 [v2:172.21.15.66:6800/163594816,v1:172.21.15.66:6801/163594816]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 27 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:02.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:03.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:02 smithi066 ceph-mon[68082]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:49:03.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:02 smithi066 ceph-mon[68082]: osd.4 [v2:172.21.15.66:6800/163594816,v1:172.21.15.66:6801/163594816] boot 2024-03-21T21:49:03.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:02 smithi066 ceph-mon[68082]: osdmap e137: 8 total, 8 up, 8 in 2024-03-21T21:49:03.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:49:03.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:03.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:03.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:02 smithi066 ceph-mon[68082]: osdmap e138: 8 total, 8 up, 8 in 2024-03-21T21:49:03.291 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[88264]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:49:03.291 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[88264]: osd.4 [v2:172.21.15.66:6800/163594816,v1:172.21.15.66:6801/163594816] boot 2024-03-21T21:49:03.291 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[88264]: osdmap e137: 8 total, 8 up, 8 in 2024-03-21T21:49:03.291 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:49:03.291 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:03.291 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:03.292 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[88264]: osdmap e138: 8 total, 8 up, 8 in 2024-03-21T21:49:03.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[85804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:49:03.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[85804]: osd.4 [v2:172.21.15.66:6800/163594816,v1:172.21.15.66:6801/163594816] boot 2024-03-21T21:49:03.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[85804]: osdmap e137: 8 total, 8 up, 8 in 2024-03-21T21:49:03.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T21:49:03.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:03.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:03.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:02 smithi028 ceph-mon[85804]: osdmap e138: 8 total, 8 up, 8 in 2024-03-21T21:49:04.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:04 smithi028 ceph-mon[85804]: pgmap v170: 161 pgs: 23 peering, 16 active+undersized+degraded, 29 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-21T21:49:04.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:04 smithi028 ceph-mon[88264]: pgmap v170: 161 pgs: 23 peering, 16 active+undersized+degraded, 29 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-21T21:49:05.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:04 smithi066 ceph-mon[68082]: pgmap v170: 161 pgs: 23 peering, 16 active+undersized+degraded, 29 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-21T21:49:06.652 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:49:06.232+0000 7f15e6bd1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-21T21:49:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: pgmap v172: 161 pgs: 38 peering, 10 active+undersized+degraded, 13 active+undersized, 100 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 28/627 objects degraded (4.466%) 2024-03-21T21:49:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:06.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:49:06.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:06.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:49:06.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:06.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:06.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:06.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:06.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 10 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: pgmap v172: 161 pgs: 38 peering, 10 active+undersized+degraded, 13 active+undersized, 100 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 28/627 objects degraded (4.466%) 2024-03-21T21:49:06.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:06.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:06.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:06.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:49:06.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:06.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:49:06.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:06.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:06.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:06.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:06.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:06 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 10 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: pgmap v172: 161 pgs: 38 peering, 10 active+undersized+degraded, 13 active+undersized, 100 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 28/627 objects degraded (4.466%) 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:07.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:06 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 10 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:07.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:07 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:07.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:07 smithi028 ceph-mon[85804]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-21T21:49:07.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:07 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:07.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:07 smithi028 ceph-mon[88264]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-21T21:49:08.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:07 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:08.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:07 smithi066 ceph-mon[68082]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-21T21:49:08.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:08 smithi028 ceph-mon[88264]: pgmap v173: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:08.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:08 smithi028 ceph-mon[88264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 10 pgs degraded) 2024-03-21T21:49:08.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:08 smithi028 ceph-mon[88264]: Cluster is now healthy 2024-03-21T21:49:08.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:08 smithi028 ceph-mon[85804]: pgmap v173: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:08.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:08 smithi028 ceph-mon[85804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 10 pgs degraded) 2024-03-21T21:49:08.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:08 smithi028 ceph-mon[85804]: Cluster is now healthy 2024-03-21T21:49:09.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:08 smithi066 ceph-mon[68082]: pgmap v173: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:09.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:08 smithi066 ceph-mon[68082]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 10 pgs degraded) 2024-03-21T21:49:09.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:08 smithi066 ceph-mon[68082]: Cluster is now healthy 2024-03-21T21:49:09.916 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:09 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:09.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:09 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:09 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:10.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:10 smithi028 ceph-mon[88264]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-21T21:49:10.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:10 smithi028 ceph-mon[85804]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-21T21:49:10.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:49:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:49:10] "GET /metrics HTTP/1.1" 200 38039 "" "Prometheus/2.43.0" 2024-03-21T21:49:11.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:10 smithi066 ceph-mon[68082]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-21T21:49:11.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:11 smithi028 ceph-mon[88264]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:49:11.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:11 smithi028 ceph-mon[85804]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:49:12.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:11 smithi066 ceph-mon[68082]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:49:14.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:13 smithi028 ceph-mon[85804]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:49:14.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:13 smithi028 ceph-mon[88264]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:49:14.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:13 smithi066 ceph-mon[68082]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T21:49:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:15 smithi028 ceph-mon[85804]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:49:16.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:15 smithi028 ceph-mon[88264]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:49:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:15 smithi066 ceph-mon[68082]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T21:49:18.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:17 smithi028 ceph-mon[85804]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:18.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:18.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:18.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:18.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:17 smithi028 ceph-mon[88264]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:18.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:18.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:18.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:18.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:17 smithi066 ceph-mon[68082]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:18.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:18.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:18.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:19.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:18 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:19.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:18 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:18 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:20.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:19 smithi028 ceph-mon[88264]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:20.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:19 smithi028 ceph-mon[85804]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:20.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:19 smithi066 ceph-mon[68082]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:21.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:49:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:49:20] "GET /metrics HTTP/1.1" 200 38189 "" "Prometheus/2.43.0" 2024-03-21T21:49:22.106 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:21 smithi066 ceph-mon[68082]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:22.106 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:22.106 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:21 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:22.106 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:21 smithi066 ceph-mon[68082]: Upgrade: osd.5 is safe to restart 2024-03-21T21:49:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:21 smithi028 ceph-mon[85804]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:21 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:21 smithi028 ceph-mon[85804]: Upgrade: osd.5 is safe to restart 2024-03-21T21:49:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:21 smithi028 ceph-mon[88264]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:21 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T21:49:22.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:21 smithi028 ceph-mon[88264]: Upgrade: osd.5 is safe to restart 2024-03-21T21:49:24.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:23 smithi066 ceph-mon[68082]: Upgrade: Updating osd.5 2024-03-21T21:49:24.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:24.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:23 smithi066 ceph-mon[68082]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:24.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T21:49:24.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:23 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:24.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:23 smithi066 ceph-mon[68082]: Deploying daemon osd.5 on smithi066 2024-03-21T21:49:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[85804]: Upgrade: Updating osd.5 2024-03-21T21:49:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[85804]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T21:49:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[85804]: Deploying daemon osd.5 on smithi066 2024-03-21T21:49:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[88264]: Upgrade: Updating osd.5 2024-03-21T21:49:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[88264]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T21:49:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:23 smithi028 ceph-mon[88264]: Deploying daemon osd.5 on smithi066 2024-03-21T21:49:26.157 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:25 smithi066 ceph-mon[68082]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:26.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:25 smithi028 ceph-mon[85804]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:26.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:25 smithi028 ceph-mon[88264]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:26.435 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:26 smithi066 systemd[1]: Stopping Ceph osd.5 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:49:26.750 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[31986]: 2024-03-21T21:49:26.645+0000 7fb364bee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:49:26.750 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[31986]: 2024-03-21T21:49:26.645+0000 7fb364bee700 -1 osd.5 138 *** Got signal Terminated *** 2024-03-21T21:49:26.751 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[31986]: 2024-03-21T21:49:26.645+0000 7fb364bee700 -1 osd.5 138 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:49:27.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:26 smithi028 ceph-mon[85804]: osd.5 marked itself down and dead 2024-03-21T21:49:27.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:26 smithi028 ceph-mon[88264]: osd.5 marked itself down and dead 2024-03-21T21:49:27.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:26 smithi066 ceph-mon[68082]: osd.5 marked itself down and dead 2024-03-21T21:49:27.672 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:27 smithi066 podman[82922]: 2024-03-21 21:49:27.373048092 +0000 UTC m=+0.858788951 container died f7aa1a1266d4518506ea1ccb1ddf0660b055fdcc0b4dc5d40c87ac42f6bd21f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_CLEAN=True, architecture=x86_64, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, version=8, io.openshift.tags=base centos centos-stream, ceph=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-21T21:49:28.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:27 smithi066 ceph-mon[68082]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:28.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:27 smithi066 ceph-mon[68082]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:49:28.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:27 smithi066 ceph-mon[68082]: osdmap e139: 8 total, 7 up, 8 in 2024-03-21T21:49:28.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:27 smithi066 podman[82922]: 2024-03-21 21:49:27.754009021 +0000 UTC m=+1.239749871 container cleanup f7aa1a1266d4518506ea1ccb1ddf0660b055fdcc0b4dc5d40c87ac42f6bd21f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5, vcs-type=git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_CLEAN=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , version=8) 2024-03-21T21:49:28.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:27 smithi066 bash[82922]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5 2024-03-21T21:49:28.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:27 smithi028 ceph-mon[88264]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:28.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:27 smithi028 ceph-mon[88264]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:49:28.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:27 smithi028 ceph-mon[88264]: osdmap e139: 8 total, 7 up, 8 in 2024-03-21T21:49:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:27 smithi028 ceph-mon[85804]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:27 smithi028 ceph-mon[85804]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:49:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:27 smithi028 ceph-mon[85804]: osdmap e139: 8 total, 7 up, 8 in 2024-03-21T21:49:28.389 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:28 smithi066 podman[82942]: 2024-03-21 21:49:28.021817135 +0000 UTC m=+0.639780747 container remove f7aa1a1266d4518506ea1ccb1ddf0660b055fdcc0b4dc5d40c87ac42f6bd21f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.buildah.version=1.19.8, ceph=True, RELEASE=HEAD, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, name=centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-03-21T21:49:28.660 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:28 smithi066 podman[83037]: 2024-03-21 21:49:28.389204328 +0000 UTC m=+0.114429133 container create bc362fe1afb5bcb341b3bd912f97f646062fd411e423426d2a1ae4f159c0fa80 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:49:28.660 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:28 smithi066 podman[83037]: 2024-03-21 21:49:28.293000943 +0000 UTC m=+0.018225768 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:49:28.968 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:28 smithi066 ceph-mon[68082]: osdmap e140: 8 total, 7 up, 8 in 2024-03-21T21:49:28.969 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:28 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:28.969 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:28 smithi066 podman[83037]: 2024-03-21 21:49:28.659037335 +0000 UTC m=+0.384262169 container init bc362fe1afb5bcb341b3bd912f97f646062fd411e423426d2a1ae4f159c0fa80 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-21T21:49:28.969 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:28 smithi066 podman[83037]: 2024-03-21 21:49:28.662768709 +0000 UTC m=+0.387993521 container start bc362fe1afb5bcb341b3bd912f97f646062fd411e423426d2a1ae4f159c0fa80 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-21T21:49:28.969 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:28 smithi066 podman[83037]: 2024-03-21 21:49:28.733447624 +0000 UTC m=+0.458672438 container attach bc362fe1afb5bcb341b3bd912f97f646062fd411e423426d2a1ae4f159c0fa80 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:49:29.165 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:49:29.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:28 smithi028 ceph-mon[88264]: osdmap e140: 8 total, 7 up, 8 in 2024-03-21T21:49:29.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:28 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:29.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:28 smithi028 ceph-mon[85804]: osdmap e140: 8 total, 7 up, 8 in 2024-03-21T21:49:29.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:28 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:29.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:28 smithi066 podman[83037]: 2024-03-21 21:49:28.968546339 +0000 UTC m=+0.693771145 container died bc362fe1afb5bcb341b3bd912f97f646062fd411e423426d2a1ae4f159c0fa80 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-21T21:49:29.836 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:49:29.836 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (7m) 49s ago 29m 27.5M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:49:29.836 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (7m) 28s ago 29m 80.9M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:49:29.836 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (6m) 49s ago 28m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:49:29.836 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (8m) 28s ago 34m 452M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:49:29.836 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (18m) 49s ago 35m 510M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:49:29.836 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (5m) 49s ago 36m 50.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:49:29.836 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (6m) 28s ago 34m 46.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:49:29.836 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (4m) 49s ago 34m 40.5M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (8m) 49s ago 30m 15.2M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (8m) 28s ago 30m 14.5M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (3m) 49s ago 33m 46.0M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (2m) 49s ago 33m 44.2M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (110s) 49s ago 33m 49.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (54s) 49s ago 32m 15.1M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (33s) 28s ago 32m 15.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ecd87bfd3243 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (31m) 28s ago 31m 56.0M 3248M 17.2.0 e1d6a67b021e f7aa1a1266d4 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (31m) 28s ago 31m 57.1M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (31m) 28s ago 31m 58.5M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (7m) 28s ago 30m 71.5M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (28m) 49s ago 28m 79.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:49:29.837 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (28m) 28s ago 28m 77.9M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:49:29.852 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:29 smithi066 podman[83064]: 2024-03-21 21:49:29.595790692 +0000 UTC m=+0.620856663 container remove bc362fe1afb5bcb341b3bd912f97f646062fd411e423426d2a1ae4f159c0fa80 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:49:29.852 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:29 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-21T21:49:29.852 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:29 smithi066 systemd[1]: Stopped Ceph osd.5 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:49:29.852 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:29 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.5.service: Consumed 17.106s CPU time. 2024-03-21T21:49:29.852 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:29 smithi066 systemd[1]: Starting Ceph osd.5 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:49:30.180 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:29 smithi066 ceph-mon[68082]: pgmap v186: 161 pgs: 7 active+undersized+degraded, 15 peering, 9 stale+active+clean, 17 active+undersized, 113 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 22/627 objects degraded (3.509%) 2024-03-21T21:49:30.180 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:29 smithi066 ceph-mon[68082]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-21T21:49:30.180 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:29 smithi066 ceph-mon[68082]: Health check failed: Degraded data redundancy: 22/627 objects degraded (3.509%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:30.180 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:29 smithi066 ceph-mon[68082]: from='client.64441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:30.180 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:29 smithi066 ceph-mon[68082]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:30.180 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 podman[83171]: 2024-03-21 21:49:29.955343039 +0000 UTC m=+0.018913513 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:49:30.180 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 podman[83171]: 2024-03-21 21:49:30.055368363 +0000 UTC m=+0.118938864 container create 147983aed254185838e4b119ad12b21d392e6e6579474d3af651418778e9db58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-21T21:49:30.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[88264]: pgmap v186: 161 pgs: 7 active+undersized+degraded, 15 peering, 9 stale+active+clean, 17 active+undersized, 113 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 22/627 objects degraded (3.509%) 2024-03-21T21:49:30.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[88264]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-21T21:49:30.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[88264]: Health check failed: Degraded data redundancy: 22/627 objects degraded (3.509%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:30.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[88264]: from='client.64441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:30.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[88264]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:30.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[85804]: pgmap v186: 161 pgs: 7 active+undersized+degraded, 15 peering, 9 stale+active+clean, 17 active+undersized, 113 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 22/627 objects degraded (3.509%) 2024-03-21T21:49:30.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[85804]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-21T21:49:30.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[85804]: Health check failed: Degraded data redundancy: 22/627 objects degraded (3.509%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:30.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[85804]: from='client.64441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:30.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:29 smithi028 ceph-mon[85804]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:30.252 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:49:30.252 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:49:30.252 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:49:30.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:49:30.252 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:49:30.252 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:49:30.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:49:30.252 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:49:30.252 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 5 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 10 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:49:30.253 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:49:30.500 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 podman[83171]: 2024-03-21 21:49:30.283012302 +0000 UTC m=+0.346582771 container init 147983aed254185838e4b119ad12b21d392e6e6579474d3af651418778e9db58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-21T21:49:30.501 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 podman[83171]: 2024-03-21 21:49:30.286935532 +0000 UTC m=+0.350506002 container start 147983aed254185838e4b119ad12b21d392e6e6579474d3af651418778e9db58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:49:30.501 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 podman[83171]: 2024-03-21 21:49:30.396027094 +0000 UTC m=+0.459597565 container attach 147983aed254185838e4b119ad12b21d392e6e6579474d3af651418778e9db58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T21:49:30.595 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:49:30.595 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:49:30.595 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:49:30.595 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T21:49:30.595 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:49:30.595 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "5/8 daemons upgraded", 2024-03-21T21:49:30.595 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T21:49:30.595 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:49:30.595 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:49:30.855 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:49:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:49:30] "GET /metrics HTTP/1.1" 200 38189 "" "Prometheus/2.43.0" 2024-03-21T21:49:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-mon[68082]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/418237340' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-mon[68082]: from='client.64465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate[83185]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate[83185]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 bash[83171]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 bash[83171]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 bash[83171]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 bash[83171]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 bash[83171]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 bash[83171]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 bash[83171]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate[83185]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate[83185]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate[83185]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-21T21:49:31.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate[83185]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-21T21:49:31.002 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate[83185]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-21T21:49:31.002 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:30 smithi066 podman[83171]: 2024-03-21 21:49:30.76583965 +0000 UTC m=+0.829410112 container died 147983aed254185838e4b119ad12b21d392e6e6579474d3af651418778e9db58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:49:31.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:30 smithi028 ceph-mon[85804]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:31.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:30 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/418237340' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:31.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:30 smithi028 ceph-mon[85804]: from='client.64465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:30 smithi028 ceph-mon[88264]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:30 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/418237340' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:30 smithi028 ceph-mon[88264]: from='client.64465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:49:31.706 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:31 smithi066 podman[83318]: 2024-03-21 21:49:31.370522281 +0000 UTC m=+0.598510505 container remove 147983aed254185838e4b119ad12b21d392e6e6579474d3af651418778e9db58 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-21T21:49:31.706 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:31 smithi066 podman[83380]: 2024-03-21 21:49:31.494532566 +0000 UTC m=+0.018601910 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:49:31.706 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:31 smithi066 podman[83380]: 2024-03-21 21:49:31.596257731 +0000 UTC m=+0.120327076 container create 08e1c86730554d8a8ed25fb9809e5b1ae6365aad19885d1d671a440a65cb2834 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-21T21:49:32.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:31 smithi066 ceph-mon[68082]: pgmap v187: 161 pgs: 12 active+undersized+degraded, 15 peering, 4 stale+active+clean, 22 active+undersized, 108 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 42/627 objects degraded (6.699%) 2024-03-21T21:49:32.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:31 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:32.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:31 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:32.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:31 smithi066 podman[83380]: 2024-03-21 21:49:31.806995582 +0000 UTC m=+0.331064935 container init 08e1c86730554d8a8ed25fb9809e5b1ae6365aad19885d1d671a440a65cb2834 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True) 2024-03-21T21:49:32.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:31 smithi066 podman[83380]: 2024-03-21 21:49:31.810183296 +0000 UTC m=+0.334252631 container start 08e1c86730554d8a8ed25fb9809e5b1ae6365aad19885d1d671a440a65cb2834 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:49:32.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:31 smithi066 ceph-osd[83399]: -- 172.21.15.66:0/2390472366 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5614efa8b680 con 0x5614efa55000 2024-03-21T21:49:32.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:31 smithi066 bash[83380]: 08e1c86730554d8a8ed25fb9809e5b1ae6365aad19885d1d671a440a65cb2834 2024-03-21T21:49:32.001 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:31 smithi066 systemd[1]: Started Ceph osd.5 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:49:32.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:31 smithi028 ceph-mon[85804]: pgmap v187: 161 pgs: 12 active+undersized+degraded, 15 peering, 4 stale+active+clean, 22 active+undersized, 108 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 42/627 objects degraded (6.699%) 2024-03-21T21:49:32.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:31 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:32.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:31 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:32.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:31 smithi028 ceph-mon[88264]: pgmap v187: 161 pgs: 12 active+undersized+degraded, 15 peering, 4 stale+active+clean, 22 active+undersized, 108 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 42/627 objects degraded (6.699%) 2024-03-21T21:49:32.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:31 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:32.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:31 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:32.750 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:49:32.363+0000 7fee2ac6c7c0 -1 Falling back to public interface 2024-03-21T21:49:33.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:33.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:33.152 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:49:33.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:33.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:33.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:49:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:49:34.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:33 smithi066 ceph-mon[68082]: pgmap v188: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-21T21:49:34.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:33 smithi066 ceph-mon[68082]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 5 pgs peering) 2024-03-21T21:49:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:33 smithi028 ceph-mon[88264]: pgmap v188: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-21T21:49:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:33 smithi028 ceph-mon[88264]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 5 pgs peering) 2024-03-21T21:49:34.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:33 smithi028 ceph-mon[85804]: pgmap v188: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-21T21:49:34.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:33 smithi028 ceph-mon[85804]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 5 pgs peering) 2024-03-21T21:49:35.500 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:49:35.094+0000 7fee2ac6c7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-21T21:49:35.501 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:49:35.132+0000 7fee2ac6c7c0 -1 osd.5 138 log_to_monitors true 2024-03-21T21:49:36.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:49:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:49:36.026+0000 7fee1c0c4700 -1 osd.5 138 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:49:36.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:36 smithi066 ceph-mon[68082]: pgmap v189: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-21T21:49:36.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:36 smithi066 ceph-mon[68082]: from='osd.5 [v2:172.21.15.66:6808/92207421,v1:172.21.15.66:6809/92207421]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:49:36.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:36 smithi066 ceph-mon[68082]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:49:36.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:36 smithi028 ceph-mon[85804]: pgmap v189: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-21T21:49:36.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:36 smithi028 ceph-mon[85804]: from='osd.5 [v2:172.21.15.66:6808/92207421,v1:172.21.15.66:6809/92207421]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:49:36.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:36 smithi028 ceph-mon[85804]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:49:36.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:36 smithi028 ceph-mon[88264]: pgmap v189: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-21T21:49:36.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:36 smithi028 ceph-mon[88264]: from='osd.5 [v2:172.21.15.66:6808/92207421,v1:172.21.15.66:6809/92207421]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:49:36.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:36 smithi028 ceph-mon[88264]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T21:49:37.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[85804]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T21:49:37.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[85804]: osdmap e141: 8 total, 7 up, 8 in 2024-03-21T21:49:37.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[85804]: from='osd.5 [v2:172.21.15.66:6808/92207421,v1:172.21.15.66:6809/92207421]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:37.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[85804]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:37.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:37.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:37.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 24 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:37.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[88264]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T21:49:37.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[88264]: osdmap e141: 8 total, 7 up, 8 in 2024-03-21T21:49:37.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[88264]: from='osd.5 [v2:172.21.15.66:6808/92207421,v1:172.21.15.66:6809/92207421]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:37.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[88264]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:37.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:37.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:37.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:37 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 24 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:37.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:37 smithi066 ceph-mon[68082]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T21:49:37.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:37 smithi066 ceph-mon[68082]: osdmap e141: 8 total, 7 up, 8 in 2024-03-21T21:49:37.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:37 smithi066 ceph-mon[68082]: from='osd.5 [v2:172.21.15.66:6808/92207421,v1:172.21.15.66:6809/92207421]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:37.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:37 smithi066 ceph-mon[68082]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:49:37.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:37 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:37.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:37 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:37.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:37 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 24 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:38.330 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:38 smithi066 ceph-mon[68082]: pgmap v191: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-21T21:49:38.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:38 smithi066 ceph-mon[68082]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:49:38.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:38 smithi066 ceph-mon[68082]: osd.5 [v2:172.21.15.66:6808/92207421,v1:172.21.15.66:6809/92207421] boot 2024-03-21T21:49:38.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:38 smithi066 ceph-mon[68082]: osdmap e142: 8 total, 8 up, 8 in 2024-03-21T21:49:38.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:38 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:49:38.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:38 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:38.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:38 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:38.331 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:38 smithi066 ceph-mon[68082]: osdmap e143: 8 total, 8 up, 8 in 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[88264]: pgmap v191: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[88264]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[88264]: osd.5 [v2:172.21.15.66:6808/92207421,v1:172.21.15.66:6809/92207421] boot 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[88264]: osdmap e142: 8 total, 8 up, 8 in 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[88264]: osdmap e143: 8 total, 8 up, 8 in 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[85804]: pgmap v191: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[85804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[85804]: osd.5 [v2:172.21.15.66:6808/92207421,v1:172.21.15.66:6809/92207421] boot 2024-03-21T21:49:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[85804]: osdmap e142: 8 total, 8 up, 8 in 2024-03-21T21:49:38.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T21:49:38.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:38.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:38.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:38 smithi028 ceph-mon[85804]: osdmap e143: 8 total, 8 up, 8 in 2024-03-21T21:49:39.315 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:39 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:39.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:39 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:39.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:39 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:40.338 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:40 smithi066 ceph-mon[68082]: pgmap v194: 161 pgs: 23 peering, 14 active+undersized+degraded, 18 active+undersized, 106 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43/627 objects degraded (6.858%) 2024-03-21T21:49:40.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:40 smithi028 ceph-mon[88264]: pgmap v194: 161 pgs: 23 peering, 14 active+undersized+degraded, 18 active+undersized, 106 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43/627 objects degraded (6.858%) 2024-03-21T21:49:40.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:40 smithi028 ceph-mon[85804]: pgmap v194: 161 pgs: 23 peering, 14 active+undersized+degraded, 18 active+undersized, 106 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43/627 objects degraded (6.858%) 2024-03-21T21:49:41.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:49:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:49:40] "GET /metrics HTTP/1.1" 200 38262 "" "Prometheus/2.43.0" 2024-03-21T21:49:41.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:49:41 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:49:41.235+0000 7f15e6bd1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-21T21:49:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: pgmap v195: 161 pgs: 23 peering, 12 active+undersized+degraded, 15 active+undersized, 111 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 37/627 objects degraded (5.901%) 2024-03-21T21:49:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:49:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:49:42.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 37/627 objects degraded (5.901%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: pgmap v195: 161 pgs: 23 peering, 12 active+undersized+degraded, 15 active+undersized, 111 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 37/627 objects degraded (5.901%) 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:49:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:49:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-21T21:49:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:42 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 37/627 objects degraded (5.901%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:42.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: pgmap v195: 161 pgs: 23 peering, 12 active+undersized+degraded, 15 active+undersized, 111 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 37/627 objects degraded (5.901%) 2024-03-21T21:49:42.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-21T21:49:42.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:42 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 37/627 objects degraded (5.901%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T21:49:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:43 smithi028 ceph-mon[88264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 37/627 objects degraded (5.901%), 12 pgs degraded) 2024-03-21T21:49:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:43 smithi028 ceph-mon[88264]: Cluster is now healthy 2024-03-21T21:49:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:43 smithi028 ceph-mon[85804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 37/627 objects degraded (5.901%), 12 pgs degraded) 2024-03-21T21:49:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:43 smithi028 ceph-mon[85804]: Cluster is now healthy 2024-03-21T21:49:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:43 smithi066 ceph-mon[68082]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 37/627 objects degraded (5.901%), 12 pgs degraded) 2024-03-21T21:49:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:43 smithi066 ceph-mon[68082]: Cluster is now healthy 2024-03-21T21:49:44.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:44 smithi028 ceph-mon[85804]: pgmap v196: 161 pgs: 13 peering, 148 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 599 B/s rd, 0 op/s 2024-03-21T21:49:44.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:44 smithi028 ceph-mon[88264]: pgmap v196: 161 pgs: 13 peering, 148 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 599 B/s rd, 0 op/s 2024-03-21T21:49:44.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:44 smithi066 ceph-mon[68082]: pgmap v196: 161 pgs: 13 peering, 148 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 599 B/s rd, 0 op/s 2024-03-21T21:49:46.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:46 smithi028 ceph-mon[85804]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:49:46.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:46 smithi028 ceph-mon[88264]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:49:46.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:46 smithi066 ceph-mon[68082]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T21:49:48.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:47 smithi028 ceph-mon[88264]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T21:49:48.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:48.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:48.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:47 smithi028 ceph-mon[85804]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T21:49:48.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:48.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:47 smithi066 ceph-mon[68082]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T21:49:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:49:49.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:48 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:49.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:48 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:49.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:48 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:50.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:49 smithi028 ceph-mon[85804]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-21T21:49:50.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:49 smithi028 ceph-mon[88264]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-21T21:49:50.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:49 smithi066 ceph-mon[68082]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-21T21:49:51.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:49:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:49:50] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.43.0" 2024-03-21T21:49:52.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:51 smithi066 ceph-mon[68082]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:52.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:51 smithi028 ceph-mon[85804]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:52.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:51 smithi028 ceph-mon[88264]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:53.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:53.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:53.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:54.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:53 smithi028 ceph-mon[85804]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:53 smithi028 ceph-mon[88264]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:54.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:53 smithi066 ceph-mon[68082]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:56.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:55 smithi028 ceph-mon[85804]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:56.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:55 smithi028 ceph-mon[88264]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:56.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:55 smithi066 ceph-mon[68082]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:49:57.134 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:56 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:57.134 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:56 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:57.134 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:56 smithi066 ceph-mon[68082]: Upgrade: osd.6 is safe to restart 2024-03-21T21:49:57.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:56 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:57.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:56 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:57.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:56 smithi028 ceph-mon[85804]: Upgrade: osd.6 is safe to restart 2024-03-21T21:49:57.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:56 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:57.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:56 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T21:49:57.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:56 smithi028 ceph-mon[88264]: Upgrade: osd.6 is safe to restart 2024-03-21T21:49:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:57 smithi066 ceph-mon[68082]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:57 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:57 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T21:49:58.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:57 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:58.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:57 smithi028 ceph-mon[85804]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:58.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:57 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:58.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:57 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T21:49:58.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:57 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:58.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:57 smithi028 ceph-mon[88264]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:49:58.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:57 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:49:58.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:57 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T21:49:58.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:57 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:49:59.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:58 smithi066 ceph-mon[68082]: Upgrade: Updating osd.6 2024-03-21T21:49:59.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:58 smithi066 ceph-mon[68082]: Deploying daemon osd.6 on smithi066 2024-03-21T21:49:59.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:58 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:59.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:58 smithi028 ceph-mon[85804]: Upgrade: Updating osd.6 2024-03-21T21:49:59.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:58 smithi028 ceph-mon[85804]: Deploying daemon osd.6 on smithi066 2024-03-21T21:49:59.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:58 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:49:59.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:58 smithi028 ceph-mon[88264]: Upgrade: Updating osd.6 2024-03-21T21:49:59.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:58 smithi028 ceph-mon[88264]: Deploying daemon osd.6 on smithi066 2024-03-21T21:49:59.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:58 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:00.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:49:59 smithi066 ceph-mon[68082]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:00.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:49:59 smithi028 ceph-mon[85804]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:00.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:49:59 smithi028 ceph-mon[88264]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:00.967 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:50:01.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:00 smithi028 ceph-mon[85804]: overall HEALTH_OK 2024-03-21T21:50:01.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:50:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:50:00] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.43.0" 2024-03-21T21:50:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:00 smithi028 ceph-mon[88264]: overall HEALTH_OK 2024-03-21T21:50:01.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:00 smithi066 ceph-mon[68082]: overall HEALTH_OK 2024-03-21T21:50:01.251 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:01 smithi066 systemd[1]: Stopping Ceph osd.6 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:50:01.642 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:50:01.642 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (8m) 80s ago 30m 27.5M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:50:01.642 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (7m) 25s ago 29m 83.0M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (7m) 80s ago 29m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (9m) 25s ago 34m 453M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (19m) 80s ago 36m 510M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (5m) 80s ago 36m 50.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (6m) 25s ago 35m 48.0M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (5m) 80s ago 35m 40.5M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (8m) 80s ago 30m 15.2M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (8m) 25s ago 30m 14.4M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (3m) 80s ago 34m 46.0M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (3m) 80s ago 34m 44.2M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (2m) 80s ago 33m 49.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (86s) 80s ago 33m 15.1M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (65s) 25s ago 32m 47.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ecd87bfd3243 2024-03-21T21:50:01.643 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (30s) 25s ago 32m 14.3M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 08e1c8673055 2024-03-21T21:50:01.644 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (32m) 25s ago 32m 57.7M 3248M 17.2.0 e1d6a67b021e 986d05ded2ea 2024-03-21T21:50:01.644 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (31m) 25s ago 31m 59.2M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:50:01.644 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (8m) 25s ago 30m 71.5M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:50:01.644 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (29m) 80s ago 29m 79.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:50:01.644 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (29m) 25s ago 29m 78.1M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:50:01.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[34879]: 2024-03-21T21:50:01.321+0000 7f023c750700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:50:01.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[34879]: 2024-03-21T21:50:01.321+0000 7f023c750700 -1 osd.6 143 *** Got signal Terminated *** 2024-03-21T21:50:01.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:01 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[34879]: 2024-03-21T21:50:01.321+0000 7f023c750700 -1 osd.6 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:50:02.074 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 6 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:50:02.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:50:02.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:50:02.076 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:50:02.076 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-21T21:50:02.076 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 11 2024-03-21T21:50:02.076 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:50:02.076 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:50:02.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:01 smithi066 ceph-mon[68082]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:02.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:01 smithi066 ceph-mon[68082]: from='client.64474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:02.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:01 smithi066 ceph-mon[68082]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:01 smithi066 ceph-mon[68082]: osd.6 marked itself down and dead 2024-03-21T21:50:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:01 smithi066 ceph-mon[68082]: from='client.64486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:02.419 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:50:02.419 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:50:02.420 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:50:02.420 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T21:50:02.420 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:50:02.420 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "6/8 daemons upgraded", 2024-03-21T21:50:02.420 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T21:50:02.420 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:50:02.420 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:50:02.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[88264]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:02.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[88264]: from='client.64474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:02.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[88264]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:02.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[88264]: osd.6 marked itself down and dead 2024-03-21T21:50:02.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[88264]: from='client.64486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:02.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:02.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[85804]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:02.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[85804]: from='client.64474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:02.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[85804]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:02.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[85804]: osd.6 marked itself down and dead 2024-03-21T21:50:02.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[85804]: from='client.64486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:02.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:02.714 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:02 smithi066 podman[87934]: 2024-03-21 21:50:02.445738986 +0000 UTC m=+1.258922988 container died 986d05ded2ea207fcad90e4bf78c1ccfaf8a0283284733a3207a916d4ecc060e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.buildah.version=1.19.8, name=centos-stream, ceph=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, release=754) 2024-03-21T21:50:03.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:02 smithi066 ceph-mon[68082]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:50:03.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:02 smithi066 ceph-mon[68082]: osdmap e144: 8 total, 7 up, 8 in 2024-03-21T21:50:03.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:02 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2025825434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:03.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:02 smithi066 ceph-mon[68082]: from='client.54561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:03.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:02 smithi066 podman[87934]: 2024-03-21 21:50:02.807839998 +0000 UTC m=+1.621024003 container cleanup 986d05ded2ea207fcad90e4bf78c1ccfaf8a0283284733a3207a916d4ecc060e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, release=754, ceph=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, version=8, architecture=x86_64, vcs-type=git, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-21T21:50:03.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:02 smithi066 bash[87934]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6 2024-03-21T21:50:03.434 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:03 smithi066 podman[87952]: 2024-03-21 21:50:03.072495505 +0000 UTC m=+0.626955925 container remove 986d05ded2ea207fcad90e4bf78c1ccfaf8a0283284733a3207a916d4ecc060e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , version=8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.buildah.version=1.19.8) 2024-03-21T21:50:03.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:02 smithi028 ceph-mon[88264]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:50:03.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:02 smithi028 ceph-mon[88264]: osdmap e144: 8 total, 7 up, 8 in 2024-03-21T21:50:03.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:02 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2025825434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:03.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:02 smithi028 ceph-mon[88264]: from='client.54561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:02 smithi028 ceph-mon[85804]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:50:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:02 smithi028 ceph-mon[85804]: osdmap e144: 8 total, 7 up, 8 in 2024-03-21T21:50:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:02 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2025825434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:02 smithi028 ceph-mon[85804]: from='client.54561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:03.685 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:03 smithi066 podman[88045]: 2024-03-21 21:50:03.337172609 +0000 UTC m=+0.018826077 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:50:03.685 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:03 smithi066 podman[88045]: 2024-03-21 21:50:03.43993191 +0000 UTC m=+0.121585350 container create b3b7c1788d1268858571f4376102e58b8a0d3b96c4f69c8ddc38fdd1cbaefdb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-21T21:50:03.962 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:03 smithi066 podman[88045]: 2024-03-21 21:50:03.68447797 +0000 UTC m=+0.366131421 container init b3b7c1788d1268858571f4376102e58b8a0d3b96c4f69c8ddc38fdd1cbaefdb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:50:03.962 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:03 smithi066 podman[88045]: 2024-03-21 21:50:03.687885408 +0000 UTC m=+0.369538851 container start b3b7c1788d1268858571f4376102e58b8a0d3b96c4f69c8ddc38fdd1cbaefdb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-21T21:50:03.962 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:03 smithi066 podman[88045]: 2024-03-21 21:50:03.767860376 +0000 UTC m=+0.449513824 container attach b3b7c1788d1268858571f4376102e58b8a0d3b96c4f69c8ddc38fdd1cbaefdb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:50:04.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:03 smithi066 ceph-mon[68082]: pgmap v207: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T21:50:04.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:03 smithi066 ceph-mon[68082]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T21:50:04.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:03 smithi066 ceph-mon[68082]: osdmap e145: 8 total, 7 up, 8 in 2024-03-21T21:50:04.251 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:03 smithi066 podman[88045]: 2024-03-21 21:50:03.987267219 +0000 UTC m=+0.668920666 container died b3b7c1788d1268858571f4376102e58b8a0d3b96c4f69c8ddc38fdd1cbaefdb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-21T21:50:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:03 smithi028 ceph-mon[85804]: pgmap v207: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T21:50:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:03 smithi028 ceph-mon[85804]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T21:50:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:03 smithi028 ceph-mon[85804]: osdmap e145: 8 total, 7 up, 8 in 2024-03-21T21:50:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:03 smithi028 ceph-mon[88264]: pgmap v207: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T21:50:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:03 smithi028 ceph-mon[88264]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T21:50:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:03 smithi028 ceph-mon[88264]: osdmap e145: 8 total, 7 up, 8 in 2024-03-21T21:50:05.000 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:04 smithi066 podman[88071]: 2024-03-21 21:50:04.688618643 +0000 UTC m=+0.692398863 container remove b3b7c1788d1268858571f4376102e58b8a0d3b96c4f69c8ddc38fdd1cbaefdb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-21T21:50:05.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:04 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-21T21:50:05.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:04 smithi066 systemd[1]: Stopped Ceph osd.6 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:50:05.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:04 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.6.service: Consumed 15.840s CPU time. 2024-03-21T21:50:05.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:04 smithi066 systemd[1]: Starting Ceph osd.6 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:50:05.265 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:05 smithi066 ceph-mon[68082]: Health check failed: Degraded data redundancy: 11/627 objects degraded (1.754%), 3 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:05.265 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 podman[88176]: 2024-03-21 21:50:05.035852145 +0000 UTC m=+0.019455444 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:50:05.266 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 podman[88176]: 2024-03-21 21:50:05.156582007 +0000 UTC m=+0.140185289 container create 5a98488a6c8ed9cddb5bf3c68f85779e163f2b9bc2bd3983c9e56c2c3292623b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-21T21:50:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:05 smithi028 ceph-mon[85804]: Health check failed: Degraded data redundancy: 11/627 objects degraded (1.754%), 3 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:05.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:05 smithi028 ceph-mon[88264]: Health check failed: Degraded data redundancy: 11/627 objects degraded (1.754%), 3 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:05.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 podman[88176]: 2024-03-21 21:50:05.36771763 +0000 UTC m=+0.351320928 container init 5a98488a6c8ed9cddb5bf3c68f85779e163f2b9bc2bd3983c9e56c2c3292623b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-21T21:50:05.751 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 podman[88176]: 2024-03-21 21:50:05.372157739 +0000 UTC m=+0.355761022 container start 5a98488a6c8ed9cddb5bf3c68f85779e163f2b9bc2bd3983c9e56c2c3292623b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-21T21:50:05.751 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 podman[88176]: 2024-03-21 21:50:05.467714948 +0000 UTC m=+0.451318248 container attach 5a98488a6c8ed9cddb5bf3c68f85779e163f2b9bc2bd3983c9e56c2c3292623b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-21T21:50:06.160 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:06 smithi066 ceph-mon[68082]: pgmap v209: 161 pgs: 3 active+undersized+degraded, 18 peering, 8 stale+active+clean, 8 active+undersized, 124 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 11/627 objects degraded (1.754%) 2024-03-21T21:50:06.160 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate[88191]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate[88191]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 bash[88176]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 bash[88176]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 bash[88176]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 bash[88176]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 bash[88176]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 bash[88176]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 bash[88176]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate[88191]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate[88191]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate[88191]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate[88191]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T21:50:06.161 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate[88191]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-21T21:50:06.162 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:05 smithi066 podman[88176]: 2024-03-21 21:50:05.855798051 +0000 UTC m=+0.839401336 container died 5a98488a6c8ed9cddb5bf3c68f85779e163f2b9bc2bd3983c9e56c2c3292623b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-21T21:50:06.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:06 smithi028 ceph-mon[85804]: pgmap v209: 161 pgs: 3 active+undersized+degraded, 18 peering, 8 stale+active+clean, 8 active+undersized, 124 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 11/627 objects degraded (1.754%) 2024-03-21T21:50:06.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:06 smithi028 ceph-mon[88264]: pgmap v209: 161 pgs: 3 active+undersized+degraded, 18 peering, 8 stale+active+clean, 8 active+undersized, 124 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 11/627 objects degraded (1.754%) 2024-03-21T21:50:06.862 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:06 smithi066 podman[88307]: 2024-03-21 21:50:06.521822127 +0000 UTC m=+0.660563950 container remove 5a98488a6c8ed9cddb5bf3c68f85779e163f2b9bc2bd3983c9e56c2c3292623b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-21T21:50:06.862 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:06 smithi066 podman[88373]: 2024-03-21 21:50:06.648323501 +0000 UTC m=+0.018535540 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:50:06.862 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:06 smithi066 podman[88373]: 2024-03-21 21:50:06.756187595 +0000 UTC m=+0.126399619 container create fbe9e137d81ceb0c250de4b01dec7ad3d06e51f43710581ba08c31134419df0a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:50:07.250 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:06 smithi066 podman[88373]: 2024-03-21 21:50:06.983820599 +0000 UTC m=+0.354032636 container init fbe9e137d81ceb0c250de4b01dec7ad3d06e51f43710581ba08c31134419df0a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-21T21:50:07.251 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:06 smithi066 podman[88373]: 2024-03-21 21:50:06.98725865 +0000 UTC m=+0.357470674 container start fbe9e137d81ceb0c250de4b01dec7ad3d06e51f43710581ba08c31134419df0a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-03-21T21:50:07.251 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:07 smithi066 ceph-osd[88392]: -- 172.21.15.66:0/3259869380 <== mon.2 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x559f21199ba0 con 0x559f2116b000 2024-03-21T21:50:07.251 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:07 smithi066 ceph-osd[88392]: -- 172.21.15.66:0/3259869380 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559f211a1680 con 0x559f2116b000 2024-03-21T21:50:07.251 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:07 smithi066 bash[88373]: fbe9e137d81ceb0c250de4b01dec7ad3d06e51f43710581ba08c31134419df0a 2024-03-21T21:50:07.251 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:07 smithi066 systemd[1]: Started Ceph osd.6 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:50:07.972 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:50:07.532+0000 7fb20a6637c0 -1 Falling back to public interface 2024-03-21T21:50:08.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:08 smithi066 ceph-mon[68082]: pgmap v210: 161 pgs: 14 active+undersized+degraded, 18 peering, 21 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%) 2024-03-21T21:50:08.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:08 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:08.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:08 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:08.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:08 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:08.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:08 smithi028 ceph-mon[85804]: pgmap v210: 161 pgs: 14 active+undersized+degraded, 18 peering, 21 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%) 2024-03-21T21:50:08.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:08 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:08.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:08 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:08.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:08 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:08.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:08 smithi028 ceph-mon[88264]: pgmap v210: 161 pgs: 14 active+undersized+degraded, 18 peering, 21 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%) 2024-03-21T21:50:08.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:08 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:08.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:08 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:08.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:08 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:09.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:09 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:09.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:09 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:09.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:09 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:10.405 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:10 smithi066 ceph-mon[68082]: pgmap v211: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-03-21T21:50:10.405 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:10 smithi066 ceph-mon[68082]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-21T21:50:10.405 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:50:10.256+0000 7fb20a6637c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-21T21:50:10.406 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:50:10.292+0000 7fb20a6637c0 -1 osd.6 143 log_to_monitors true 2024-03-21T21:50:10.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:10 smithi028 ceph-mon[88264]: pgmap v211: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-03-21T21:50:10.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:10 smithi028 ceph-mon[88264]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-21T21:50:10.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:10 smithi028 ceph-mon[85804]: pgmap v211: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-03-21T21:50:10.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:10 smithi028 ceph-mon[85804]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-21T21:50:11.151 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:50:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:50:10] "GET /metrics HTTP/1.1" 200 38270 "" "Prometheus/2.43.0" 2024-03-21T21:50:11.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:11 smithi066 ceph-mon[68082]: from='osd.6 [v2:172.21.15.66:6816/356915899,v1:172.21.15.66:6817/356915899]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:50:11.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:11 smithi066 ceph-mon[68082]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:50:11.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:11 smithi028 ceph-mon[85804]: from='osd.6 [v2:172.21.15.66:6816/356915899,v1:172.21.15.66:6817/356915899]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:50:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:11 smithi028 ceph-mon[85804]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:50:11.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:11 smithi028 ceph-mon[88264]: from='osd.6 [v2:172.21.15.66:6816/356915899,v1:172.21.15.66:6817/356915899]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:50:11.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:11 smithi028 ceph-mon[88264]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T21:50:12.000 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:50:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:50:11.641+0000 7fb1f7ab3700 -1 osd.6 143 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[88264]: pgmap v212: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[88264]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[88264]: osdmap e146: 8 total, 7 up, 8 in 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[88264]: from='osd.6 [v2:172.21.15.66:6816/356915899,v1:172.21.15.66:6817/356915899]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[88264]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[88264]: from='osd.6 ' entity='osd.6' 2024-03-21T21:50:12.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[85804]: pgmap v212: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-03-21T21:50:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[85804]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T21:50:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[85804]: osdmap e146: 8 total, 7 up, 8 in 2024-03-21T21:50:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[85804]: from='osd.6 [v2:172.21.15.66:6816/356915899,v1:172.21.15.66:6817/356915899]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[85804]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:12 smithi028 ceph-mon[85804]: from='osd.6 ' entity='osd.6' 2024-03-21T21:50:12.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:12 smithi066 ceph-mon[68082]: pgmap v212: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-03-21T21:50:12.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:12 smithi066 ceph-mon[68082]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T21:50:12.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:12 smithi066 ceph-mon[68082]: osdmap e146: 8 total, 7 up, 8 in 2024-03-21T21:50:12.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:12 smithi066 ceph-mon[68082]: from='osd.6 [v2:172.21.15.66:6816/356915899,v1:172.21.15.66:6817/356915899]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:12.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:12 smithi066 ceph-mon[68082]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:12.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:12 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:12.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:12 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:12.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:12 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 77/627 objects degraded (12.281%), 20 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:12.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:12 smithi066 ceph-mon[68082]: from='osd.6 ' entity='osd.6' 2024-03-21T21:50:13.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[88264]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:50:13.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[88264]: osd.6 [v2:172.21.15.66:6816/356915899,v1:172.21.15.66:6817/356915899] boot 2024-03-21T21:50:13.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[88264]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T21:50:13.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:50:13.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:13.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:13.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[85804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:50:13.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[85804]: osd.6 [v2:172.21.15.66:6816/356915899,v1:172.21.15.66:6817/356915899] boot 2024-03-21T21:50:13.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[85804]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T21:50:13.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:50:13.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:13.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:13 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:13.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:13 smithi066 ceph-mon[68082]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:50:13.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:13 smithi066 ceph-mon[68082]: osd.6 [v2:172.21.15.66:6816/356915899,v1:172.21.15.66:6817/356915899] boot 2024-03-21T21:50:13.489 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:13 smithi066 ceph-mon[68082]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T21:50:13.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T21:50:13.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:13.490 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:13 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:14.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:14 smithi028 ceph-mon[88264]: pgmap v215: 161 pgs: 9 peering, 17 active+undersized+degraded, 27 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-03-21T21:50:14.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:14 smithi028 ceph-mon[88264]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T21:50:14.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:14 smithi028 ceph-mon[85804]: pgmap v215: 161 pgs: 9 peering, 17 active+undersized+degraded, 27 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-03-21T21:50:14.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:14 smithi028 ceph-mon[85804]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T21:50:14.955 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:14 smithi066 ceph-mon[68082]: pgmap v215: 161 pgs: 9 peering, 17 active+undersized+degraded, 27 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-03-21T21:50:14.955 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:14 smithi066 ceph-mon[68082]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T21:50:16.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:50:16.585+0000 7f15e6bd1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-03-21T21:50:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: pgmap v217: 161 pgs: 14 peering, 13 active+undersized+degraded, 16 active+undersized, 118 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 42/627 objects degraded (6.699%) 2024-03-21T21:50:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 42/627 objects degraded (6.699%), 13 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:16.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: pgmap v217: 161 pgs: 14 peering, 13 active+undersized+degraded, 16 active+undersized, 118 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 42/627 objects degraded (6.699%) 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 42/627 objects degraded (6.699%), 13 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:16.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:16.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:17.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: pgmap v217: 161 pgs: 14 peering, 13 active+undersized+degraded, 16 active+undersized, 118 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 42/627 objects degraded (6.699%) 2024-03-21T21:50:17.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 42/627 objects degraded (6.699%), 13 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:17.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:17.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:17 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:17 smithi028 ceph-mon[88264]: Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-03-21T21:50:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:17 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:17 smithi028 ceph-mon[85804]: Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-03-21T21:50:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:18.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:17 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:18.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:17 smithi066 ceph-mon[68082]: Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-03-21T21:50:18.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:18.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:18 smithi028 ceph-mon[88264]: pgmap v218: 161 pgs: 14 peering, 147 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T21:50:18.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:18 smithi028 ceph-mon[88264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/627 objects degraded (6.699%), 13 pgs degraded) 2024-03-21T21:50:18.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:18 smithi028 ceph-mon[88264]: Cluster is now healthy 2024-03-21T21:50:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:18 smithi028 ceph-mon[85804]: pgmap v218: 161 pgs: 14 peering, 147 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T21:50:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:18 smithi028 ceph-mon[85804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/627 objects degraded (6.699%), 13 pgs degraded) 2024-03-21T21:50:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:18 smithi028 ceph-mon[85804]: Cluster is now healthy 2024-03-21T21:50:19.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:18 smithi066 ceph-mon[68082]: pgmap v218: 161 pgs: 14 peering, 147 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T21:50:19.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:18 smithi066 ceph-mon[68082]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/627 objects degraded (6.699%), 13 pgs degraded) 2024-03-21T21:50:19.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:18 smithi066 ceph-mon[68082]: Cluster is now healthy 2024-03-21T21:50:19.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:19 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:19.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:19 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:20.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:19 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:20.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:20 smithi028 ceph-mon[88264]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 399 B/s rd, 0 op/s 2024-03-21T21:50:20.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:20 smithi028 ceph-mon[85804]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 399 B/s rd, 0 op/s 2024-03-21T21:50:20.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:50:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:50:20] "GET /metrics HTTP/1.1" 200 38275 "" "Prometheus/2.43.0" 2024-03-21T21:50:21.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:20 smithi066 ceph-mon[68082]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 399 B/s rd, 0 op/s 2024-03-21T21:50:22.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:22 smithi028 ceph-mon[88264]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-03-21T21:50:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:22 smithi028 ceph-mon[85804]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-03-21T21:50:23.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:22 smithi066 ceph-mon[68082]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-03-21T21:50:24.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:24 smithi028 ceph-mon[88264]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-21T21:50:24.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:24 smithi028 ceph-mon[85804]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-21T21:50:25.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:24 smithi066 ceph-mon[68082]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-21T21:50:26.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:26 smithi028 ceph-mon[88264]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:26.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:26 smithi028 ceph-mon[85804]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:27.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:26 smithi066 ceph-mon[68082]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:28.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:27 smithi066 ceph-mon[68082]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:28.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:27 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:28.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:27 smithi028 ceph-mon[85804]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:28.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:27 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:28.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:27 smithi028 ceph-mon[88264]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:28.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:27 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:29.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:28 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:29.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:28 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:29.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:28 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:30.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:29 smithi066 ceph-mon[68082]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:30.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:29 smithi028 ceph-mon[88264]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:30.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:29 smithi028 ceph-mon[85804]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:31.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:50:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:50:30] "GET /metrics HTTP/1.1" 200 38275 "" "Prometheus/2.43.0" 2024-03-21T21:50:32.220 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:31 smithi066 ceph-mon[68082]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:32.220 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:31 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:32.220 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:31 smithi066 ceph-mon[68082]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:32.220 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:31 smithi066 ceph-mon[68082]: Upgrade: osd.7 is safe to restart 2024-03-21T21:50:32.220 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:31 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:32.220 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:31 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:32.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[88264]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:32.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:32.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[88264]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:32.362 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[88264]: Upgrade: osd.7 is safe to restart 2024-03-21T21:50:32.362 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:32.362 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:32.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[85804]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:32.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:32.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[85804]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T21:50:32.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[85804]: Upgrade: osd.7 is safe to restart 2024-03-21T21:50:32.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:32.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:31 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:32.785 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (8m) 112s ago 30m 27.5M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (8m) 21s ago 30m 82.8M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (7m) 112s ago 29m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (9m) 21s ago 35m 453M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (19m) 112s ago 37m 510M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (6m) 112s ago 37m 50.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (7m) 21s ago 35m 49.2M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (6m) 112s ago 35m 40.5M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (9m) 112s ago 31m 15.2M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:50:33.477 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (9m) 21s ago 31m 14.3M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (4m) 112s ago 35m 46.0M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (3m) 112s ago 34m 44.2M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (2m) 112s ago 34m 49.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (117s) 112s ago 33m 15.1M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (97s) 21s ago 33m 48.5M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ecd87bfd3243 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (61s) 21s ago 33m 43.7M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 08e1c8673055 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (26s) 21s ago 32m 12.7M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 fbe9e137d81c 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (32m) 21s ago 32m 60.5M 3248M 17.2.0 e1d6a67b021e 5d95a40446f8 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (8m) 21s ago 31m 71.5M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (29m) 112s ago 29m 79.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:50:33.478 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (29m) 21s ago 29m 78.3M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:50:33.890 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:50:33.890 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:50:33.890 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:50:33.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:50:33.890 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:50:33.890 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:50:33.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 7 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 12 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:50:33.891 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:50:34.233 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:50:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:50:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:50:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T21:50:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:50:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "7/8 daemons upgraded", 2024-03-21T21:50:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T21:50:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:50:34.234 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:50:34.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: from='client.64510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:34.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:34.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: from='client.64516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:34.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: Upgrade: Updating osd.7 2024-03-21T21:50:34.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:34.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T21:50:34.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:34.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: Deploying daemon osd.7 on smithi066 2024-03-21T21:50:34.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:34.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:34 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/3687721593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: from='client.64510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: from='client.64516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: Upgrade: Updating osd.7 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: Deploying daemon osd.7 on smithi066 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:34.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/3687721593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: from='client.64510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: from='client.64516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: Upgrade: Updating osd.7 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: Deploying daemon osd.7 on smithi066 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:34 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/3687721593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:35.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:35 smithi066 ceph-mon[68082]: from='client.54597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:35.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:35 smithi028 ceph-mon[85804]: from='client.54597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:35.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:35 smithi028 ceph-mon[88264]: from='client.54597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:50:36.563 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:36 smithi066 ceph-mon[68082]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:36.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:36 smithi028 ceph-mon[85804]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:36.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:36 smithi028 ceph-mon[88264]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:50:36.989 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:36 smithi066 systemd[1]: Stopping Ceph osd.7 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:50:37.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[37801]: 2024-03-21T21:50:37.041+0000 7f0913472700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:50:37.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[37801]: 2024-03-21T21:50:37.041+0000 7f0913472700 -1 osd.7 148 *** Got signal Terminated *** 2024-03-21T21:50:37.251 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[37801]: 2024-03-21T21:50:37.041+0000 7f0913472700 -1 osd.7 148 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:50:37.621 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:37 smithi066 ceph-mon[68082]: osd.7 marked itself down and dead 2024-03-21T21:50:37.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:37 smithi028 ceph-mon[85804]: osd.7 marked itself down and dead 2024-03-21T21:50:37.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:37 smithi028 ceph-mon[88264]: osd.7 marked itself down and dead 2024-03-21T21:50:37.938 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:37 smithi066 podman[92906]: 2024-03-21 21:50:37.621123506 +0000 UTC m=+0.720732683 container died 5d95a40446f82730f7ac408d577bbef22ca9c500769bac822c3ee3bad0019b88 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, architecture=x86_64, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, name=centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8) 2024-03-21T21:50:38.246 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:38 smithi066 podman[92906]: 2024-03-21 21:50:38.052941082 +0000 UTC m=+1.152550254 container cleanup 5d95a40446f82730f7ac408d577bbef22ca9c500769bac822c3ee3bad0019b88 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, RELEASE=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.tags=base centos centos-stream) 2024-03-21T21:50:38.246 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:38 smithi066 bash[92906]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7 2024-03-21T21:50:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:38 smithi028 ceph-mon[88264]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-21T21:50:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:38 smithi028 ceph-mon[88264]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:50:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:38 smithi028 ceph-mon[88264]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-21T21:50:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:38 smithi028 ceph-mon[88264]: osdmap e149: 8 total, 7 up, 8 in 2024-03-21T21:50:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:38 smithi028 ceph-mon[85804]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-21T21:50:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:38 smithi028 ceph-mon[85804]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:50:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:38 smithi028 ceph-mon[85804]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-21T21:50:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:38 smithi028 ceph-mon[85804]: osdmap e149: 8 total, 7 up, 8 in 2024-03-21T21:50:38.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:38 smithi066 ceph-mon[68082]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-21T21:50:38.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:38 smithi066 ceph-mon[68082]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T21:50:38.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:38 smithi066 ceph-mon[68082]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-21T21:50:38.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:38 smithi066 ceph-mon[68082]: osdmap e149: 8 total, 7 up, 8 in 2024-03-21T21:50:38.500 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:38 smithi066 podman[92933]: 2024-03-21 21:50:38.245773682 +0000 UTC m=+0.622629202 container remove 5d95a40446f82730f7ac408d577bbef22ca9c500769bac822c3ee3bad0019b88 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, RELEASE=HEAD, architecture=x86_64, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, ceph=True, distribution-scope=public, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream) 2024-03-21T21:50:38.942 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:38 smithi066 podman[93028]: 2024-03-21 21:50:38.587017955 +0000 UTC m=+0.019214875 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:50:38.942 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:38 smithi066 podman[93028]: 2024-03-21 21:50:38.713425567 +0000 UTC m=+0.145622488 container create f3956fe858b0efafe47280a252c1a1aa464c2331ebd9092344b22dd24335052a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-21T21:50:39.251 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:38 smithi066 podman[93028]: 2024-03-21 21:50:38.941388916 +0000 UTC m=+0.373585844 container init f3956fe858b0efafe47280a252c1a1aa464c2331ebd9092344b22dd24335052a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-21T21:50:39.251 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:38 smithi066 podman[93028]: 2024-03-21 21:50:38.945297746 +0000 UTC m=+0.377494669 container start f3956fe858b0efafe47280a252c1a1aa464c2331ebd9092344b22dd24335052a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-21T21:50:39.251 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:39 smithi066 podman[93028]: 2024-03-21 21:50:39.024358888 +0000 UTC m=+0.456555823 container attach f3956fe858b0efafe47280a252c1a1aa464c2331ebd9092344b22dd24335052a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:50:39.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:39 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:39.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:39 smithi028 ceph-mon[88264]: osdmap e150: 8 total, 7 up, 8 in 2024-03-21T21:50:39.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:39 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:39.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:39 smithi028 ceph-mon[85804]: osdmap e150: 8 total, 7 up, 8 in 2024-03-21T21:50:39.716 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:39 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:39.716 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:39 smithi066 ceph-mon[68082]: osdmap e150: 8 total, 7 up, 8 in 2024-03-21T21:50:39.717 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:39 smithi066 podman[93028]: 2024-03-21 21:50:39.260668954 +0000 UTC m=+0.692865868 container died f3956fe858b0efafe47280a252c1a1aa464c2331ebd9092344b22dd24335052a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-03-21T21:50:40.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:39 smithi066 podman[93055]: 2024-03-21 21:50:39.795187403 +0000 UTC m=+0.529528904 container remove f3956fe858b0efafe47280a252c1a1aa464c2331ebd9092344b22dd24335052a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5) 2024-03-21T21:50:40.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:39 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-21T21:50:40.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:39 smithi066 systemd[1]: Stopped Ceph osd.7 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:50:40.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:39 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.7.service: Consumed 18.652s CPU time. 2024-03-21T21:50:40.296 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:40 smithi066 systemd[1]: Starting Ceph osd.7 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:50:40.297 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:40 smithi066 podman[93160]: 2024-03-21 21:50:40.152643162 +0000 UTC m=+0.018894818 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:50:40.297 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:40 smithi066 podman[93160]: 2024-03-21 21:50:40.263175658 +0000 UTC m=+0.129427313 container create 94725211469deaf0ab54b9c132af80e73374c73da8f053fffe5894cc83fd7cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:50:40.558 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:40 smithi066 ceph-mon[68082]: pgmap v231: 161 pgs: 1 active+undersized+degraded, 17 peering, 16 stale+active+clean, 6 active+undersized, 121 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 44 op/s; 1/627 objects degraded (0.159%) 2024-03-21T21:50:40.558 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:40 smithi066 ceph-mon[68082]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-21T21:50:40.558 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:40 smithi066 ceph-mon[68082]: Health check failed: Degraded data redundancy: 1/627 objects degraded (0.159%), 1 pg degraded (PG_DEGRADED) 2024-03-21T21:50:40.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:40 smithi028 ceph-mon[88264]: pgmap v231: 161 pgs: 1 active+undersized+degraded, 17 peering, 16 stale+active+clean, 6 active+undersized, 121 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 44 op/s; 1/627 objects degraded (0.159%) 2024-03-21T21:50:40.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:40 smithi028 ceph-mon[88264]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-21T21:50:40.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:40 smithi028 ceph-mon[88264]: Health check failed: Degraded data redundancy: 1/627 objects degraded (0.159%), 1 pg degraded (PG_DEGRADED) 2024-03-21T21:50:40.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:40 smithi028 ceph-mon[85804]: pgmap v231: 161 pgs: 1 active+undersized+degraded, 17 peering, 16 stale+active+clean, 6 active+undersized, 121 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 44 op/s; 1/627 objects degraded (0.159%) 2024-03-21T21:50:40.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:40 smithi028 ceph-mon[85804]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-21T21:50:40.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:40 smithi028 ceph-mon[85804]: Health check failed: Degraded data redundancy: 1/627 objects degraded (0.159%), 1 pg degraded (PG_DEGRADED) 2024-03-21T21:50:41.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:40 smithi066 podman[93160]: 2024-03-21 21:50:40.557700283 +0000 UTC m=+0.423951940 container init 94725211469deaf0ab54b9c132af80e73374c73da8f053fffe5894cc83fd7cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-21T21:50:41.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:40 smithi066 podman[93160]: 2024-03-21 21:50:40.561142052 +0000 UTC m=+0.427393713 container start 94725211469deaf0ab54b9c132af80e73374c73da8f053fffe5894cc83fd7cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:50:41.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:40 smithi066 podman[93160]: 2024-03-21 21:50:40.632593165 +0000 UTC m=+0.498844836 container attach 94725211469deaf0ab54b9c132af80e73374c73da8f053fffe5894cc83fd7cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-21T21:50:41.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:50:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:50:40] "GET /metrics HTTP/1.1" 200 38280 "" "Prometheus/2.43.0" 2024-03-21T21:50:41.328 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate[93175]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-21T21:50:41.328 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate[93175]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-21T21:50:41.328 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 bash[93160]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-21T21:50:41.328 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 bash[93160]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-21T21:50:41.328 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 bash[93160]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-21T21:50:41.329 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 bash[93160]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-21T21:50:41.329 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 bash[93160]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-21T21:50:41.329 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 bash[93160]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-21T21:50:41.329 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 bash[93160]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-21T21:50:41.329 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate[93175]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-21T21:50:41.329 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate[93175]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-21T21:50:41.329 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate[93175]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-21T21:50:41.329 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate[93175]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-21T21:50:41.329 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate[93175]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-21T21:50:41.330 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 podman[93160]: 2024-03-21 21:50:41.046536766 +0000 UTC m=+0.912788425 container died 94725211469deaf0ab54b9c132af80e73374c73da8f053fffe5894cc83fd7cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:50:41.978 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 podman[93302]: 2024-03-21 21:50:41.619700174 +0000 UTC m=+0.567925595 container remove 94725211469deaf0ab54b9c132af80e73374c73da8f053fffe5894cc83fd7cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:50:41.978 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 podman[93368]: 2024-03-21 21:50:41.7431338 +0000 UTC m=+0.019333250 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:50:41.978 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:41 smithi066 podman[93368]: 2024-03-21 21:50:41.862313312 +0000 UTC m=+0.138512749 container create ededc79acc52d19540667b0828c49ad633673b60d127d681fe9fc2da3d2dde64 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:50:42.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:42 smithi066 podman[93368]: 2024-03-21 21:50:42.164771683 +0000 UTC m=+0.440971133 container init ededc79acc52d19540667b0828c49ad633673b60d127d681fe9fc2da3d2dde64 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T21:50:42.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:42 smithi066 podman[93368]: 2024-03-21 21:50:42.168325281 +0000 UTC m=+0.444524719 container start ededc79acc52d19540667b0828c49ad633673b60d127d681fe9fc2da3d2dde64 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2) 2024-03-21T21:50:42.251 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:42 smithi066 ceph-osd[93388]: -- 172.21.15.66:0/3544049218 <== mon.2 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5620c00b1680 con 0x5620c007b400 2024-03-21T21:50:42.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:42 smithi028 ceph-mon[85804]: pgmap v232: 161 pgs: 6 active+undersized+degraded, 17 peering, 15 stale+active+clean, 14 active+undersized, 109 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 62 op/s; 19/627 objects degraded (3.030%) 2024-03-21T21:50:42.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:42 smithi028 ceph-mon[88264]: pgmap v232: 161 pgs: 6 active+undersized+degraded, 17 peering, 15 stale+active+clean, 14 active+undersized, 109 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 62 op/s; 19/627 objects degraded (3.030%) 2024-03-21T21:50:42.713 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:42 smithi066 ceph-mon[68082]: pgmap v232: 161 pgs: 6 active+undersized+degraded, 17 peering, 15 stale+active+clean, 14 active+undersized, 109 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 62 op/s; 19/627 objects degraded (3.030%) 2024-03-21T21:50:42.713 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:42 smithi066 bash[93368]: ededc79acc52d19540667b0828c49ad633673b60d127d681fe9fc2da3d2dde64 2024-03-21T21:50:42.713 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:42 smithi066 systemd[1]: Started Ceph osd.7 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:50:43.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:42 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:50:42.712+0000 7fed49fec7c0 -1 Falling back to public interface 2024-03-21T21:50:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:43 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:43 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:43.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:43 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:43.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:43 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:43.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:43 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:43.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:43 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:43 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:43 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:43 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:44.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:44 smithi066 ceph-mon[68082]: pgmap v233: 161 pgs: 19 active+undersized+degraded, 17 peering, 33 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 85 op/s; 69/627 objects degraded (11.005%) 2024-03-21T21:50:44.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:44 smithi028 ceph-mon[88264]: pgmap v233: 161 pgs: 19 active+undersized+degraded, 17 peering, 33 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 85 op/s; 69/627 objects degraded (11.005%) 2024-03-21T21:50:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:44 smithi028 ceph-mon[85804]: pgmap v233: 161 pgs: 19 active+undersized+degraded, 17 peering, 33 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 85 op/s; 69/627 objects degraded (11.005%) 2024-03-21T21:50:45.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:45 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:45.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:45 smithi028 ceph-mon[88264]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-21T21:50:45.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:45 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:45.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:45 smithi028 ceph-mon[85804]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-21T21:50:45.724 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:45 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:45.724 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:45 smithi066 ceph-mon[68082]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-21T21:50:45.724 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:45 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:50:45.435+0000 7fed49fec7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-21T21:50:45.724 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:45 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:50:45.465+0000 7fed49fec7c0 -1 osd.7 148 log_to_monitors true 2024-03-21T21:50:46.558 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:46 smithi066 ceph-mon[68082]: pgmap v234: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 98 op/s; 80/627 objects degraded (12.759%) 2024-03-21T21:50:46.558 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:46 smithi066 ceph-mon[68082]: from='osd.7 [v2:172.21.15.66:6824/354175520,v1:172.21.15.66:6825/354175520]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:50:46.558 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:46 smithi066 ceph-mon[68082]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:50:46.558 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:50:46 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:50:46.400+0000 7fed3b444700 -1 osd.7 148 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T21:50:46.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:46 smithi028 ceph-mon[88264]: pgmap v234: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 98 op/s; 80/627 objects degraded (12.759%) 2024-03-21T21:50:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:46 smithi028 ceph-mon[88264]: from='osd.7 [v2:172.21.15.66:6824/354175520,v1:172.21.15.66:6825/354175520]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:50:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:46 smithi028 ceph-mon[88264]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:50:46.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:46 smithi028 ceph-mon[85804]: pgmap v234: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 98 op/s; 80/627 objects degraded (12.759%) 2024-03-21T21:50:46.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:46 smithi028 ceph-mon[85804]: from='osd.7 [v2:172.21.15.66:6824/354175520,v1:172.21.15.66:6825/354175520]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:50:46.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:46 smithi028 ceph-mon[85804]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[88264]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[88264]: osdmap e151: 8 total, 7 up, 8 in 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[88264]: from='osd.7 [v2:172.21.15.66:6824/354175520,v1:172.21.15.66:6825/354175520]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[88264]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[85804]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T21:50:47.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[85804]: osdmap e151: 8 total, 7 up, 8 in 2024-03-21T21:50:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[85804]: from='osd.7 [v2:172.21.15.66:6824/354175520,v1:172.21.15.66:6825/354175520]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[85804]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:47.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:47 smithi066 ceph-mon[68082]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T21:50:47.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:47 smithi066 ceph-mon[68082]: osdmap e151: 8 total, 7 up, 8 in 2024-03-21T21:50:47.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:47 smithi066 ceph-mon[68082]: from='osd.7 [v2:172.21.15.66:6824/354175520,v1:172.21.15.66:6825/354175520]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:47.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:47 smithi066 ceph-mon[68082]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-21T21:50:47.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:47.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:47.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:47.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:50:48.652 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:48 smithi066 ceph-mon[68082]: pgmap v236: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 112 op/s; 80/627 objects degraded (12.759%) 2024-03-21T21:50:48.652 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:48 smithi066 ceph-mon[68082]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:50:48.652 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:48 smithi066 ceph-mon[68082]: osd.7 [v2:172.21.15.66:6824/354175520,v1:172.21.15.66:6825/354175520] boot 2024-03-21T21:50:48.652 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:48 smithi066 ceph-mon[68082]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T21:50:48.652 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:50:48.652 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:48.652 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:48.652 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:48 smithi066 ceph-mon[68082]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T21:50:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[88264]: pgmap v236: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 112 op/s; 80/627 objects degraded (12.759%) 2024-03-21T21:50:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[88264]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:50:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[88264]: osd.7 [v2:172.21.15.66:6824/354175520,v1:172.21.15.66:6825/354175520] boot 2024-03-21T21:50:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[88264]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T21:50:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:50:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[88264]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T21:50:48.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[85804]: pgmap v236: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 112 op/s; 80/627 objects degraded (12.759%) 2024-03-21T21:50:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[85804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T21:50:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[85804]: osd.7 [v2:172.21.15.66:6824/354175520,v1:172.21.15.66:6825/354175520] boot 2024-03-21T21:50:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[85804]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T21:50:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T21:50:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:48 smithi028 ceph-mon[85804]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T21:50:49.649 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:49 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:49.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:49 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:49.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:49 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:50.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:50 smithi028 ceph-mon[85804]: pgmap v239: 161 pgs: 17 peering, 20 active+undersized+degraded, 32 active+undersized, 92 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 71/627 objects degraded (11.324%) 2024-03-21T21:50:50.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:50 smithi028 ceph-mon[88264]: pgmap v239: 161 pgs: 17 peering, 20 active+undersized+degraded, 32 active+undersized, 92 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 71/627 objects degraded (11.324%) 2024-03-21T21:50:50.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:50 smithi066 ceph-mon[68082]: pgmap v239: 161 pgs: 17 peering, 20 active+undersized+degraded, 32 active+undersized, 92 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 71/627 objects degraded (11.324%) 2024-03-21T21:50:51.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:50:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:50:50] "GET /metrics HTTP/1.1" 200 38218 "" "Prometheus/2.43.0" 2024-03-21T21:50:51.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:51 smithi066 ceph-mon[68082]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 16 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:51.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:51 smithi028 ceph-mon[85804]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 16 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:51.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:51 smithi028 ceph-mon[88264]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 16 pgs degraded (PG_DEGRADED) 2024-03-21T21:50:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: pgmap v240: 161 pgs: 17 peering, 16 active+undersized+degraded, 24 active+undersized, 104 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s; 57/627 objects degraded (9.091%); 0 B/s, 1 objects/s recovering 2024-03-21T21:50:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-21T21:50:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: pgmap v240: 161 pgs: 17 peering, 16 active+undersized+degraded, 24 active+undersized, 104 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s; 57/627 objects degraded (9.091%); 0 B/s, 1 objects/s recovering 2024-03-21T21:50:52.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-21T21:50:52.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-21T21:50:52.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-21T21:50:53.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: pgmap v240: 161 pgs: 17 peering, 16 active+undersized+degraded, 24 active+undersized, 104 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s; 57/627 objects degraded (9.091%); 0 B/s, 1 objects/s recovering 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-21T21:50:53.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-21T21:50:53.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-21T21:50:53.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-21T21:50:53.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-21T21:50:53.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-21T21:50:53.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-21T21:50:53.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-21T21:50:53.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all osd 2024-03-21T21:50:53.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: Upgrade: Setting require_osd_release to 19 squid 2024-03-21T21:50:53.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-21T21:50:53.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-21T21:50:53.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T21:50:53.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all osd 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: Upgrade: Setting require_osd_release to 19 squid 2024-03-21T21:50:53.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:50:53.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:50:53.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:50:53.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:53.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:53.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:53 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all osd 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: Upgrade: Setting require_osd_release to 19 squid 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:50:54.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:50:54.003 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:50:54.004 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:53 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:54.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mds 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all nfs 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: Upgrade: Finalizing container_image settings 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: Upgrade: Complete! 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: pgmap v242: 161 pgs: 17 peering, 144 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 14 op/s; 0 B/s, 7 objects/s recovering 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 16 pgs degraded) 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[88264]: Cluster is now healthy 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mds 2024-03-21T21:50:54.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:50:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:50:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all nfs 2024-03-21T21:50:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:50:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: Upgrade: Finalizing container_image settings 2024-03-21T21:50:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: Upgrade: Complete! 2024-03-21T21:50:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: pgmap v242: 161 pgs: 17 peering, 144 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 14 op/s; 0 B/s, 7 objects/s recovering 2024-03-21T21:50:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 16 pgs degraded) 2024-03-21T21:50:54.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:54 smithi028 ceph-mon[85804]: Cluster is now healthy 2024-03-21T21:50:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mds 2024-03-21T21:50:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:50:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:50:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nfs 2024-03-21T21:50:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:50:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: Upgrade: Finalizing container_image settings 2024-03-21T21:50:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: Upgrade: Complete! 2024-03-21T21:50:55.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: pgmap v242: 161 pgs: 17 peering, 144 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 14 op/s; 0 B/s, 7 objects/s recovering 2024-03-21T21:50:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 16 pgs degraded) 2024-03-21T21:50:55.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:54 smithi066 ceph-mon[68082]: Cluster is now healthy 2024-03-21T21:50:56.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:56 smithi028 ceph-mon[85804]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 6 objects/s recovering 2024-03-21T21:50:56.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:56 smithi028 ceph-mon[88264]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 6 objects/s recovering 2024-03-21T21:50:57.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:56 smithi066 ceph-mon[68082]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 6 objects/s recovering 2024-03-21T21:50:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:57 smithi066 ceph-mon[68082]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-03-21T21:50:58.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:57 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:58.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:57 smithi028 ceph-mon[85804]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-03-21T21:50:58.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:57 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:58.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:57 smithi028 ceph-mon[88264]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-03-21T21:50:58.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:57 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:50:59.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:58 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:59.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:58 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:50:59.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:58 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:00.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:50:59 smithi066 ceph-mon[68082]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-03-21T21:51:00.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:50:59 smithi028 ceph-mon[85804]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-03-21T21:51:00.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:50:59 smithi028 ceph-mon[88264]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-03-21T21:51:01.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:51:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:51:00] "GET /metrics HTTP/1.1" 200 38218 "" "Prometheus/2.43.0" 2024-03-21T21:51:02.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:01 smithi066 ceph-mon[68082]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-21T21:51:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:02.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:01 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:02.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:01 smithi028 ceph-mon[88264]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-21T21:51:02.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:02.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:01 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:01 smithi028 ceph-mon[85804]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-21T21:51:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:02.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:01 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:03.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:03.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:04.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:03 smithi066 ceph-mon[68082]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T21:51:04.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:03 smithi028 ceph-mon[88264]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T21:51:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:03 smithi028 ceph-mon[85804]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T21:51:05.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:04 smithi066 ceph-mon[68082]: from='client.54609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:05.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:04 smithi028 ceph-mon[88264]: from='client.54609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:05.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:04 smithi028 ceph-mon[85804]: from='client.54609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:05.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:51:06.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:05 smithi066 ceph-mon[68082]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T21:51:06.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:05 smithi028 ceph-mon[88264]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T21:51:06.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:05 smithi028 ceph-mon[85804]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T21:51:07.629 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:51:07.629 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (9m) 2m ago 31m 27.5M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:51:07.629 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (8m) 20s ago 30m 82.8M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:51:07.629 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (8m) 2m ago 30m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:51:07.629 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (10m) 20s ago 35m 453M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:51:07.629 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (20m) 2m ago 37m 510M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:51:07.629 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (7m) 2m ago 37m 50.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:51:07.629 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (7m) 20s ago 36m 48.5M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:51:07.629 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (6m) 2m ago 36m 40.5M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (9m) 2m ago 32m 15.2M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (9m) 20s ago 31m 14.3M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (4m) 2m ago 35m 46.0M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (4m) 2m ago 35m 44.2M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (3m) 2m ago 34m 49.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (2m) 2m ago 34m 15.1M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (2m) 20s ago 34m 50.7M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ecd87bfd3243 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (96s) 20s ago 33m 45.8M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 08e1c8673055 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (60s) 20s ago 33m 51.9M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 fbe9e137d81c 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (25s) 20s ago 32m 12.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ededc79acc52 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (9m) 20s ago 32m 71.5M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (30m) 2m ago 30m 79.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:51:07.630 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (30m) 20s ago 30m 78.6M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:51:08.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:07 smithi028 ceph-mon[85804]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:08.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:07 smithi028 ceph-mon[85804]: from='client.54615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:08.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:07 smithi028 ceph-mon[88264]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:08.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:07 smithi028 ceph-mon[88264]: from='client.54615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:08.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:07 smithi066 ceph-mon[68082]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:08.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:07 smithi066 ceph-mon[68082]: from='client.54615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:08.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-03-21T21:51:09.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:08 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:09.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:08 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:09.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:08 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:10.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:09 smithi028 ceph-mon[88264]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:10.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:09 smithi028 ceph-mon[85804]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:10.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:09 smithi066 ceph-mon[68082]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:10.585 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:51:10.880 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:51:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:51:10] "GET /metrics HTTP/1.1" 200 38371 "" "Prometheus/2.43.0" 2024-03-21T21:51:11.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:10 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/4193772909' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:11.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:10 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/4193772909' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:11.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:10 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/4193772909' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:11.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-03-21T21:51:12.850 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:12 smithi028 ceph-mon[88264]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:12.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:12 smithi028 ceph-mon[85804]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:13.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:12 smithi066 ceph-mon[68082]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:13.492 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)" 2024-03-21T21:51:13.805 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:13 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2200689999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:13.806 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:13 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2200689999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:14.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:13 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2200689999' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:14.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade status' 2024-03-21T21:51:14.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:14 smithi028 ceph-mon[85804]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:14.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:14 smithi028 ceph-mon[88264]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:15.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:14 smithi066 ceph-mon[68082]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:16.232 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:51:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": null, 2024-03-21T21:51:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": false, 2024-03-21T21:51:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "", 2024-03-21T21:51:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:51:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": null, 2024-03-21T21:51:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-03-21T21:51:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:51:16.233 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:51:16.863 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:16 smithi028 ceph-mon[88264]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:16.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:16 smithi028 ceph-mon[85804]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:16.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:51:17.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:16 smithi066 ceph-mon[68082]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:17 smithi028 ceph-mon[88264]: from='client.54633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:17 smithi028 ceph-mon[85804]: from='client.54633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:18.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:17 smithi066 ceph-mon[68082]: from='client.54633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:18.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:18.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:18 smithi028 ceph-mon[88264]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:18.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:18 smithi028 ceph-mon[85804]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:19.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:18 smithi066 ceph-mon[68082]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:19.036 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-03-21T21:51:19.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:19 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:19.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:19 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/3185962938' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:51:19.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:19 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:19.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:19 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/3185962938' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:51:19.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-03-21T21:51:20.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:19 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:20.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:19 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/3185962938' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:51:20.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:20 smithi028 ceph-mon[85804]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:20.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:20 smithi028 ceph-mon[88264]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:20.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:51:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:51:20] "GET /metrics HTTP/1.1" 200 38370 "" "Prometheus/2.43.0" 2024-03-21T21:51:21.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:20 smithi066 ceph-mon[68082]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:22.877 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:22 smithi028 ceph-mon[88264]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:22.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:22 smithi028 ceph-mon[85804]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:23.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:22 smithi066 ceph-mon[68082]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:23.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:23 smithi028 ceph-mon[88264]: from='client.54645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:23.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:23 smithi028 ceph-mon[85804]: from='client.54645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:23.918 INFO:teuthology.orchestra.run.smithi028.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:51:24.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:23 smithi066 ceph-mon[68082]: from='client.54645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:24.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T21:51:24.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[88264]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[85804]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:24.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:24 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:25.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:24 smithi066 ceph-mon[68082]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:25.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:24 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:25.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:24 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:25.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:24 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:25.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:24 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:25.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:24 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:25.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:25 smithi028 ceph-mon[88264]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:51:25.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:25 smithi028 ceph-mon[85804]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:51:26.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:25 smithi066 ceph-mon[68082]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:51:26.621 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:51:26.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:26 smithi028 ceph-mon[88264]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:51:26.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:26 smithi028 ceph-mon[88264]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:26.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:26 smithi028 ceph-mon[85804]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:51:26.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:26 smithi028 ceph-mon[85804]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:27.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:26 smithi066 ceph-mon[68082]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:51:27.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:26 smithi066 ceph-mon[68082]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:27.311 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:51:27.332 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (9m) 2m ago 31m 27.5M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:51:27.332 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (9m) 40s ago 31m 82.8M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:51:27.332 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (8m) 2m ago 30m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:51:27.332 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (10m) 40s ago 36m 453M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (20m) 2m ago 37m 510M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (7m) 2m ago 38m 50.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (8m) 40s ago 36m 48.5M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (6m) 2m ago 36m 40.5M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (10m) 2m ago 32m 15.2M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (10m) 40s ago 32m 14.3M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (5m) 2m ago 35m 46.0M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (4m) 2m ago 35m 44.2M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (3m) 2m ago 35m 49.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (2m) 2m ago 34m 15.1M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (2m) 40s ago 34m 50.7M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ecd87bfd3243 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (115s) 40s ago 33m 45.8M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 08e1c8673055 2024-03-21T21:51:27.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (80s) 40s ago 33m 51.9M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 fbe9e137d81c 2024-03-21T21:51:27.334 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (45s) 40s ago 33m 12.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ededc79acc52 2024-03-21T21:51:27.334 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (9m) 40s ago 32m 71.5M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:51:27.334 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (30m) 2m ago 30m 79.1M - 17.2.0 e1d6a67b021e 0812f586db80 2024-03-21T21:51:27.334 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (30m) 40s ago 30m 78.6M - 17.2.0 e1d6a67b021e e67fe12363ba 2024-03-21T21:51:27.740 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:51:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:51:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:51:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:51:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 8 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 13 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:51:27.741 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:51:27.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:27 smithi028 ceph-mon[88264]: from='client.54651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:27.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:27 smithi028 ceph-mon[85804]: from='client.54651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:28.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:27 smithi066 ceph-mon[68082]: from='client.54651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:28.093 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:51:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:51:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:51:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-03-21T21:51:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:51:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "0/2 daemons upgraded", 2024-03-21T21:51:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading rgw daemons", 2024-03-21T21:51:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:51:28.094 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:51:28.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:28.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='client.64594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='client.54663 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mgr 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mon 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all crash 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all osd 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mds 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2776293062' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='client.64594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='client.54663 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:28.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mgr 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mon 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all crash 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all osd 2024-03-21T21:51:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:28.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mds 2024-03-21T21:51:28.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:28.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:28 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2776293062' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:29.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:29.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='client.64594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:29.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='client.54663 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mgr 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mon 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all crash 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:29.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:29.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all osd 2024-03-21T21:51:29.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:29.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:29.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mds 2024-03-21T21:51:29.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:29.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:28 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2776293062' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:29.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[85804]: from='client.54675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:29.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:29.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:29.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:51:29.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[88264]: from='client.54675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:51:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:29 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:29 smithi066 ceph-mon[68082]: from='client.54675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:29 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:29 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:29 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi028.qubqiy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:51:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:29 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:30.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:30 smithi028 ceph-mon[85804]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:30.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:30 smithi028 ceph-mon[85804]: Upgrade: Updating rgw.foo.smithi028.qubqiy (1/2) 2024-03-21T21:51:30.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:30 smithi028 ceph-mon[85804]: Deploying daemon rgw.foo.smithi028.qubqiy on smithi028 2024-03-21T21:51:30.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:51:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:51:30] "GET /metrics HTTP/1.1" 200 38370 "" "Prometheus/2.43.0" 2024-03-21T21:51:30.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:30 smithi028 ceph-mon[88264]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:30.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:30 smithi028 ceph-mon[88264]: Upgrade: Updating rgw.foo.smithi028.qubqiy (1/2) 2024-03-21T21:51:30.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:30 smithi028 ceph-mon[88264]: Deploying daemon rgw.foo.smithi028.qubqiy on smithi028 2024-03-21T21:51:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:30 smithi066 ceph-mon[68082]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:30 smithi066 ceph-mon[68082]: Upgrade: Updating rgw.foo.smithi028.qubqiy (1/2) 2024-03-21T21:51:31.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:30 smithi066 ceph-mon[68082]: Deploying daemon rgw.foo.smithi028.qubqiy on smithi028 2024-03-21T21:51:32.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:32 smithi028 ceph-mon[88264]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:32.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:32.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:32 smithi028 ceph-mon[85804]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:32.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:33.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:32 smithi066 ceph-mon[68082]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:51:33.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:34.634 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:34 smithi028 ceph-mon[88264]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:34.634 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:34 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:34.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:34 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:34.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:34 smithi028 ceph-mon[85804]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:34.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:34 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:34.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:34 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:34.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:34 smithi066 ceph-mon[68082]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:51:34.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:34 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:34.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:34 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:36.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:36 smithi066 ceph-mon[68082]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T21:51:36.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:36 smithi066 ceph-mon[68082]: Upgrade: Updating rgw.foo.smithi066.gkaqro (2/2) 2024-03-21T21:51:36.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:36.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:51:36.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:36.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:36 smithi066 ceph-mon[68082]: Deploying daemon rgw.foo.smithi066.gkaqro on smithi066 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[85804]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[85804]: Upgrade: Updating rgw.foo.smithi066.gkaqro (2/2) 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[85804]: Deploying daemon rgw.foo.smithi066.gkaqro on smithi066 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[88264]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[88264]: Upgrade: Updating rgw.foo.smithi066.gkaqro (2/2) 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:36.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi066.gkaqro", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T21:51:36.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:36.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:36 smithi028 ceph-mon[88264]: Deploying daemon rgw.foo.smithi066.gkaqro on smithi066 2024-03-21T21:51:38.360 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:38 smithi066 ceph-mon[68082]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-21T21:51:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:38 smithi028 ceph-mon[85804]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-21T21:51:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:38 smithi028 ceph-mon[88264]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-21T21:51:39.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:39 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:39.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:39.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:39.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:39 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:39.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:39 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:39.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:39.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:39.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:39 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:39.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:39 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:39.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:39.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:39.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:39 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:40.074 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:40 smithi066 ceph-mon[68082]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-21T21:51:40.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:40 smithi028 ceph-mon[88264]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-21T21:51:40.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:40 smithi028 ceph-mon[85804]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-21T21:51:41.099 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:51:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:51:40] "GET /metrics HTTP/1.1" 200 38366 "" "Prometheus/2.43.0" 2024-03-21T21:51:42.274 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:42 smithi066 ceph-mon[68082]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 85 B/s wr, 98 op/s 2024-03-21T21:51:42.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:42 smithi028 ceph-mon[88264]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 85 B/s wr, 98 op/s 2024-03-21T21:51:42.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:42 smithi028 ceph-mon[85804]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 85 B/s wr, 98 op/s 2024-03-21T21:51:44.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[85804]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 142 op/s 2024-03-21T21:51:44.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:43 smithi028 ceph-mon[88264]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 142 op/s 2024-03-21T21:51:44.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:43 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:43 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:43 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:43 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:44.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:43 smithi066 ceph-mon[68082]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 142 op/s 2024-03-21T21:51:45.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:44 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:44 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:44 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:44 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:44 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:44 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:44 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:44 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:44 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:44 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:44 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:45.251 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:44 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:46.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:46 smithi028 ceph-mon[85804]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-21T21:51:46.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:46 smithi028 ceph-mon[88264]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-21T21:51:46.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:46 smithi066 ceph-mon[68082]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-21T21:51:47.155 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:47.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:47.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:51:48.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 153 op/s 2024-03-21T21:51:48.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:48.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.752 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi028.qubqiy"}]: dispatch 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi028.qubqiy"}]': finished 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi066.gkaqro"}]: dispatch 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi066.gkaqro"}]': finished 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.753 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.754 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:51:48.755 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:48.756 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:48.757 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:48.757 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.757 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:48.757 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:48.757 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:48.757 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:48 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 153 op/s 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi028.qubqiy"}]: dispatch 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi028.qubqiy"}]': finished 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi066.gkaqro"}]: dispatch 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi066.gkaqro"}]': finished 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 153 op/s 2024-03-21T21:51:48.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi028.qubqiy"}]: dispatch 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi028.qubqiy"}]': finished 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi066.gkaqro"}]: dispatch 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi066.gkaqro"}]': finished 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:51:48.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T21:51:48.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:51:48.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:48.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:51:48.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:48 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:49.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:49 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all rgw 2024-03-21T21:51:49.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:49 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:51:49.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:49 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:51:49.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:49 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nfs 2024-03-21T21:51:49.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:49 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:51:49.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:49 smithi066 ceph-mon[68082]: Upgrade: Finalizing container_image settings 2024-03-21T21:51:49.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:49 smithi066 ceph-mon[68082]: Upgrade: Complete! 2024-03-21T21:51:49.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:49 smithi066 ceph-mon[68082]: Checking dashboard <-> RGW credentials 2024-03-21T21:51:49.751 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:49 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all rgw 2024-03-21T21:51:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:51:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:51:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all nfs 2024-03-21T21:51:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:51:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[85804]: Upgrade: Finalizing container_image settings 2024-03-21T21:51:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[85804]: Upgrade: Complete! 2024-03-21T21:51:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[85804]: Checking dashboard <-> RGW credentials 2024-03-21T21:51:49.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all rgw 2024-03-21T21:51:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:51:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:51:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all nfs 2024-03-21T21:51:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all nvmeof 2024-03-21T21:51:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[88264]: Upgrade: Finalizing container_image settings 2024-03-21T21:51:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[88264]: Upgrade: Complete! 2024-03-21T21:51:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[88264]: Checking dashboard <-> RGW credentials 2024-03-21T21:51:49.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:49 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:50.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:50 smithi066 ceph-mon[68082]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 170 B/s wr, 109 op/s 2024-03-21T21:51:50.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:50 smithi028 ceph-mon[88264]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 170 B/s wr, 109 op/s 2024-03-21T21:51:50.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:50 smithi028 ceph-mon[85804]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 170 B/s wr, 109 op/s 2024-03-21T21:51:51.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:51:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:51:50] "GET /metrics HTTP/1.1" 200 38404 "" "Prometheus/2.43.0" 2024-03-21T21:51:52.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:52 smithi066 ceph-mon[68082]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 85 op/s 2024-03-21T21:51:52.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:52 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:52.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:52 smithi028 ceph-mon[85804]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 85 op/s 2024-03-21T21:51:52.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:52 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:52.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:52 smithi028 ceph-mon[88264]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 85 op/s 2024-03-21T21:51:52.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:52 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:51:54.750 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:54 smithi066 ceph-mon[68082]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 85 B/s wr, 72 op/s 2024-03-21T21:51:54.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:54 smithi028 ceph-mon[85804]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 85 B/s wr, 72 op/s 2024-03-21T21:51:54.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:54 smithi028 ceph-mon[88264]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 85 B/s wr, 72 op/s 2024-03-21T21:51:56.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:56 smithi028 ceph-mon[88264]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 85 B/s wr, 28 op/s 2024-03-21T21:51:56.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:56 smithi028 ceph-mon[85804]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 85 B/s wr, 28 op/s 2024-03-21T21:51:57.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:56 smithi066 ceph-mon[68082]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 85 B/s wr, 28 op/s 2024-03-21T21:51:58.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:58 smithi028 ceph-mon[88264]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-21T21:51:58.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:58 smithi028 ceph-mon[85804]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-21T21:51:59.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:58 smithi066 ceph-mon[68082]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-21T21:51:59.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:51:59.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:59 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:59.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:51:59 smithi028 ceph-mon[88264]: from='client.64696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:51:59.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:59 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:51:59.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:51:59 smithi028 ceph-mon[85804]: from='client.64696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:00.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:59 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:00.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:51:59 smithi066 ceph-mon[68082]: from='client.64696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:00.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:00 smithi028 ceph-mon[88264]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-21T21:52:00.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:00 smithi028 ceph-mon[85804]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-21T21:52:00.839 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:52:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:52:00] "GET /metrics HTTP/1.1" 200 38404 "" "Prometheus/2.43.0" 2024-03-21T21:52:01.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:00 smithi066 ceph-mon[68082]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-21T21:52:01.286 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:52:01.286 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (10m) 18s ago 32m 27.7M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (9m) 18s ago 31m 83.1M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (9m) 18s ago 31m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (11m) 18s ago 36m 454M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (21m) 18s ago 38m 515M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (7m) 18s ago 38m 58.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (8m) 18s ago 37m 51.7M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (7m) 18s ago 37m 46.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (10m) 18s ago 32m 14.9M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (10m) 18s ago 32m 14.6M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (5m) 18s ago 36m 54.5M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (5m) 18s ago 36m 52.3M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (4m) 18s ago 35m 55.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (3m) 18s ago 35m 61.7M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (3m) 18s ago 34m 54.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ecd87bfd3243 2024-03-21T21:52:01.287 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (2m) 18s ago 34m 48.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 08e1c8673055 2024-03-21T21:52:01.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (114s) 18s ago 34m 54.5M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 fbe9e137d81c 2024-03-21T21:52:01.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (79s) 18s ago 33m 47.2M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ededc79acc52 2024-03-21T21:52:01.288 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (10m) 18s ago 32m 71.5M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:52:01.288 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (28s) 18s ago 31m 75.4M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 08cdd63331d3 2024-03-21T21:52:01.288 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (22s) 18s ago 31m 75.2M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 b65365b5be02 2024-03-21T21:52:01.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-03-21T21:52:02.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:02 smithi028 ceph-mon[88264]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-21T21:52:02.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:02 smithi028 ceph-mon[88264]: from='client.54759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:02.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:02.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:02 smithi028 ceph-mon[85804]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-21T21:52:02.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:02 smithi028 ceph-mon[85804]: from='client.54759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:02.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:03.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:02 smithi066 ceph-mon[68082]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-21T21:52:03.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:02 smithi066 ceph-mon[68082]: from='client.54759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:03.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:03.971 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:52:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:04 smithi028 ceph-mon[88264]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-03-21T21:52:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:04 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/1273906752' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:04.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:04 smithi028 ceph-mon[85804]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-03-21T21:52:04.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:04 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/1273906752' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:04.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-03-21T21:52:05.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:04 smithi066 ceph-mon[68082]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-03-21T21:52:05.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:04 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/1273906752' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:06 smithi028 ceph-mon[85804]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T21:52:06.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:06 smithi028 ceph-mon[88264]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T21:52:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)" 2024-03-21T21:52:07.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:06 smithi066 ceph-mon[68082]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T21:52:07.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade status' 2024-03-21T21:52:07.919 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:07 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/4210806758' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:07.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:07 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/4210806758' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:08.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:07 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/4210806758' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:08.891 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:08 smithi028 ceph-mon[88264]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:08.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:08 smithi028 ceph-mon[85804]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:09.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:08 smithi066 ceph-mon[68082]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:09.603 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:52:09.604 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": null, 2024-03-21T21:52:09.604 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": false, 2024-03-21T21:52:09.604 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "", 2024-03-21T21:52:09.604 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:52:09.604 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": null, 2024-03-21T21:52:09.604 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-03-21T21:52:09.604 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:52:09.604 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:52:09.901 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:09 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:09 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:09.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:09 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:10.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:52:10.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:10 smithi028 ceph-mon[88264]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:10.812 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:10 smithi028 ceph-mon[88264]: from='client.64720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:10.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:10 smithi028 ceph-mon[85804]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:10.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:10 smithi028 ceph-mon[85804]: from='client.64720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:11.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:10 smithi066 ceph-mon[68082]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:11.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:10 smithi066 ceph-mon[68082]: from='client.64720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:11.140 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:52:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:52:10] "GET /metrics HTTP/1.1" 200 38404 "" "Prometheus/2.43.0" 2024-03-21T21:52:12.280 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-03-21T21:52:12.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:12 smithi028 ceph-mon[88264]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:12.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:12 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/1556967048' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:52:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:12 smithi028 ceph-mon[85804]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:12 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/1556967048' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:52:12.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-21T21:52:13.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:12 smithi066 ceph-mon[68082]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:13.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:12 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/1556967048' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:52:14.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:14 smithi028 ceph-mon[85804]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:14.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:14 smithi028 ceph-mon[88264]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:14.898 INFO:teuthology.orchestra.run.smithi028.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:52:15.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:14 smithi066 ceph-mon[68082]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:15.603 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-21T21:52:15.619 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T21:52:15.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[85804]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[85804]: from='client.64732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[85804]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[85804]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:52:16.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[88264]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:16.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[88264]: from='client.64732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:16.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[88264]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:52:16.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:16.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:52:16.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:52:16.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:52:16.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:16.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:15 smithi028 ceph-mon[88264]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:52:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:15 smithi066 ceph-mon[68082]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:15 smithi066 ceph-mon[68082]: from='client.64732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:15 smithi066 ceph-mon[68082]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:52:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:15 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:15 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:52:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:15 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:52:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:15 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:52:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:15 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:16.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:15 smithi066 ceph-mon[68082]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 2024-03-21T21:52:17.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:16 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:17.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:16 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:17.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:16 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:17.587 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:52:18.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:17 smithi028 ceph-mon[88264]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:18.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:17 smithi028 ceph-mon[88264]: from='client.64738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:18.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:17 smithi028 ceph-mon[85804]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:18.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:17 smithi028 ceph-mon[85804]: from='client.64738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:18.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:17 smithi066 ceph-mon[68082]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:18.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:17 smithi066 ceph-mon[68082]: from='client.64738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (10m) 35s ago 32m 27.7M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (9m) 35s ago 32m 83.1M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (9m) 35s ago 31m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (11m) 35s ago 37m 454M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (21m) 35s ago 38m 515M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (8m) 35s ago 39m 58.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (9m) 35s ago 37m 51.7M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (7m) 35s ago 37m 46.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (11m) 35s ago 33m 14.9M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (10m) 35s ago 33m 14.6M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (6m) 35s ago 36m 54.5M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:52:18.263 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (5m) 35s ago 36m 52.3M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:52:18.264 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (4m) 35s ago 35m 55.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:52:18.264 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (3m) 35s ago 35m 61.7M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:52:18.264 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (3m) 35s ago 35m 54.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ecd87bfd3243 2024-03-21T21:52:18.264 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (2m) 35s ago 34m 48.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 08e1c8673055 2024-03-21T21:52:18.264 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (2m) 35s ago 34m 54.5M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 fbe9e137d81c 2024-03-21T21:52:18.264 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (96s) 35s ago 34m 47.2M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ededc79acc52 2024-03-21T21:52:18.264 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (10m) 35s ago 33m 71.5M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:52:18.264 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (45s) 35s ago 31m 75.4M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 08cdd63331d3 2024-03-21T21:52:18.264 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (39s) 35s ago 31m 75.2M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 b65365b5be02 2024-03-21T21:52:18.672 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:52:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:52:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:52:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 8 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 15 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:52:18.673 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:52:18.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:18 smithi028 ceph-mon[88264]: from='client.64744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:18.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:18 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:18.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:18 smithi028 ceph-mon[88264]: from='client.64750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:18.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:18 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2679205609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:18 smithi028 ceph-mon[85804]: from='client.64744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:18 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:18 smithi028 ceph-mon[85804]: from='client.64750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:18.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:18 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2679205609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:19.030 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:52:19.030 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38", 2024-03-21T21:52:19.030 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:52:19.030 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-21T21:52:19.031 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-03-21T21:52:19.031 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "", 2024-03-21T21:52:19.031 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38 image", 2024-03-21T21:52:19.031 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-03-21T21:52:19.031 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:52:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:18 smithi066 ceph-mon[68082]: from='client.64744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:18 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:18 smithi066 ceph-mon[68082]: from='client.64750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:19.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:18 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2679205609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:19.451 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-03-21T21:52:20.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:19 smithi028 ceph-mon[85804]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:20.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:19 smithi028 ceph-mon[85804]: from='client.64762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:20.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:19 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/3654488441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:52:20.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:19 smithi028 ceph-mon[88264]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:20.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:19 smithi028 ceph-mon[88264]: from='client.64762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:20.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:19 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/3654488441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:52:20.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:19 smithi066 ceph-mon[68082]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:20.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:19 smithi066 ceph-mon[68082]: from='client.64762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:20.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:19 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/3654488441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:52:21.000 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:52:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=cleanup t=2024-03-21T21:52:20.637271192Z level=info msg="Completed cleanup jobs" duration=1.084981ms 2024-03-21T21:52:21.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:52:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:52:20] "GET /metrics HTTP/1.1" 200 38401 "" "Prometheus/2.43.0" 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mgr 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mon 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all crash 2024-03-21T21:52:21.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all osd 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all mds 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all rgw 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:52:21.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mgr 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mon 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all crash 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all osd 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all mds 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all rgw 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:52:21.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:52:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:52:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:21 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Target is version 19.0.0-2286-g5d87ad5b (squid) 2024-03-21T21:52:22.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d'] 2024-03-21T21:52:22.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mgr 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mon 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all crash 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all osd 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all mds 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.001 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all rgw 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T21:52:22.002 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:21 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:22 smithi028 ceph-mon[85804]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:22.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:22 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:22 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:52:22.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:22 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:52:22.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:22 smithi028 ceph-mon[88264]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:22.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:22 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:22.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:22 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:52:22.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:22 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:52:23.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:22 smithi066 ceph-mon[68082]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:23.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:22 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:23.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:22 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi028.lrtsfq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T21:52:23.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:22 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:52:24.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:23 smithi066 ceph-mon[68082]: Upgrade: Updating iscsi.foo.smithi028.lrtsfq 2024-03-21T21:52:24.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:23 smithi066 ceph-mon[68082]: Deploying daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:52:24.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:23 smithi028 ceph-mon[88264]: Upgrade: Updating iscsi.foo.smithi028.lrtsfq 2024-03-21T21:52:24.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:23 smithi028 ceph-mon[88264]: Deploying daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:52:24.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:23 smithi028 ceph-mon[85804]: Upgrade: Updating iscsi.foo.smithi028.lrtsfq 2024-03-21T21:52:24.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:23 smithi028 ceph-mon[85804]: Deploying daemon iscsi.foo.smithi028.lrtsfq on smithi028 2024-03-21T21:52:25.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:24 smithi066 ceph-mon[68082]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:25.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:24 smithi028 ceph-mon[85804]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:25.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:24 smithi028 ceph-mon[88264]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:26.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:25 smithi066 ceph-mon[68082]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:26.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:25 smithi028 ceph-mon[88264]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:25 smithi028 ceph-mon[85804]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:28.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:27 smithi028 ceph-mon[88264]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:27 smithi028 ceph-mon[85804]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:28.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:27 smithi066 ceph-mon[68082]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:28 smithi028 ceph-mon[88264]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:29.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:28 smithi028 ceph-mon[85804]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:29.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:28 smithi066 ceph-mon[68082]: from='client.25600 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:30.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:29 smithi028 ceph-mon[88264]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:30.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:29 smithi028 ceph-mon[85804]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:30.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:29 smithi066 ceph-mon[68082]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:31.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:52:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:52:30] "GET /metrics HTTP/1.1" 200 38401 "" "Prometheus/2.43.0" 2024-03-21T21:52:32.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:31 smithi066 ceph-mon[68082]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:32.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:31 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:32.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:31 smithi028 ceph-mon[88264]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:32.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:31 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:32.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:31 smithi028 ceph-mon[85804]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:32.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:31 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:34.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:33 smithi066 ceph-mon[68082]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:34.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:33 smithi028 ceph-mon[85804]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:34.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:33 smithi028 ceph-mon[88264]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:52:36.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:35 smithi028 ceph-mon[88264]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:36.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:35 smithi028 ceph-mon[85804]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:36.217 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:52:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:52:35.968+0000 7f15b7369700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed. 2024-03-21T21:52:36.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:35 smithi066 ceph-mon[68082]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:52:37.080 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[85804]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed. 2024-03-21T21:52:37.080 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:37.080 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[85804]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[88264]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed. 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[88264]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:52:37.081 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:37.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:36 smithi066 ceph-mon[68082]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed. 2024-03-21T21:52:37.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:37.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:52:37.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:36 smithi066 ceph-mon[68082]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-21T21:52:37.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:52:37.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:52:37.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:38.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:37 smithi066 ceph-mon[68082]: Health check failed: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-21T21:52:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:37 smithi028 ceph-mon[85804]: Health check failed: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-21T21:52:38.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:37 smithi028 ceph-mon[88264]: Health check failed: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-21T21:52:39.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:38 smithi066 ceph-mon[68082]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-21T21:52:39.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:38 smithi028 ceph-mon[85804]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-21T21:52:39.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:38 smithi028 ceph-mon[88264]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-21T21:52:41.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:40 smithi028 ceph-mon[85804]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-21T21:52:41.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:40 smithi028 ceph-mon[88264]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-21T21:52:41.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:52:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:52:40] "GET /metrics HTTP/1.1" 200 38401 "" "Prometheus/2.43.0" 2024-03-21T21:52:41.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:40 smithi066 ceph-mon[68082]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-21T21:52:43.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:42 smithi066 ceph-mon[68082]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 461 B/s rd, 0 op/s 2024-03-21T21:52:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:42 smithi028 ceph-mon[85804]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 461 B/s rd, 0 op/s 2024-03-21T21:52:43.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:42 smithi028 ceph-mon[88264]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 461 B/s rd, 0 op/s 2024-03-21T21:52:45.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:44 smithi066 ceph-mon[68082]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 461 B/s rd, 0 op/s 2024-03-21T21:52:45.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:44 smithi028 ceph-mon[85804]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 461 B/s rd, 0 op/s 2024-03-21T21:52:45.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:44 smithi028 ceph-mon[88264]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 461 B/s rd, 0 op/s 2024-03-21T21:52:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:46 smithi066 ceph-mon[68082]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:52:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:47.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:46 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:47.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:46 smithi028 ceph-mon[85804]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:52:47.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:47.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:46 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:47.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:46 smithi028 ceph-mon[88264]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:52:47.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:52:47.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:46 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:52:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:47 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/3787851189' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:52:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:47 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/1631385480' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/1625650397"}]: dispatch 2024-03-21T21:52:48.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:47 smithi066 ceph-mon[68082]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/1625650397"}]: dispatch 2024-03-21T21:52:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:47 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/3787851189' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:52:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:47 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/1631385480' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/1625650397"}]: dispatch 2024-03-21T21:52:48.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:47 smithi028 ceph-mon[88264]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/1625650397"}]: dispatch 2024-03-21T21:52:48.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:47 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/3787851189' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T21:52:48.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:47 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/1631385480' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/1625650397"}]: dispatch 2024-03-21T21:52:48.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:47 smithi028 ceph-mon[85804]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/1625650397"}]: dispatch 2024-03-21T21:52:49.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:49 smithi028 ceph-mon[85804]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:52:49.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:49 smithi028 ceph-mon[85804]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/1625650397"}]': finished 2024-03-21T21:52:49.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:49 smithi028 ceph-mon[85804]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T21:52:49.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:49 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/881323516' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1125666882"}]: dispatch 2024-03-21T21:52:49.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:49 smithi028 ceph-mon[88264]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:52:49.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:49 smithi028 ceph-mon[88264]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/1625650397"}]': finished 2024-03-21T21:52:49.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:49 smithi028 ceph-mon[88264]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T21:52:49.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:49 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/881323516' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1125666882"}]: dispatch 2024-03-21T21:52:49.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:49 smithi066 ceph-mon[68082]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:52:49.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:49 smithi066 ceph-mon[68082]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6801/1625650397"}]': finished 2024-03-21T21:52:49.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:49 smithi066 ceph-mon[68082]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T21:52:49.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:49 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/881323516' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1125666882"}]: dispatch 2024-03-21T21:52:49.813 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:52:50.163 INFO:teuthology.orchestra.run.smithi028.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed." 2024-03-21T21:52:50.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:50 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/881323516' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1125666882"}]': finished 2024-03-21T21:52:50.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:50 smithi028 ceph-mon[88264]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T21:52:50.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:50 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/85029403' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/1625650397"}]: dispatch 2024-03-21T21:52:50.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:50 smithi028 ceph-mon[88264]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/1625650397"}]: dispatch 2024-03-21T21:52:50.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:50 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/881323516' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1125666882"}]': finished 2024-03-21T21:52:50.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:50 smithi028 ceph-mon[85804]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T21:52:50.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:50 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/85029403' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/1625650397"}]: dispatch 2024-03-21T21:52:50.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:50 smithi028 ceph-mon[85804]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/1625650397"}]: dispatch 2024-03-21T21:52:50.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:50 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/881323516' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1125666882"}]': finished 2024-03-21T21:52:50.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:50 smithi066 ceph-mon[68082]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T21:52:50.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:50 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/85029403' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/1625650397"}]: dispatch 2024-03-21T21:52:50.250 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:50 smithi066 ceph-mon[68082]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/1625650397"}]: dispatch 2024-03-21T21:52:50.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:52:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:52:50] "GET /metrics HTTP/1.1" 200 38477 "" "Prometheus/2.43.0" 2024-03-21T21:52:51.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:52:51.338 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[88264]: from='client.54873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:51.338 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[88264]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-03-21T21:52:51.338 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[88264]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/1625650397"}]': finished 2024-03-21T21:52:51.338 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[88264]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T21:52:51.338 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[88264]: from='client.54879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:51.339 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2628290879' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1937719450"}]: dispatch 2024-03-21T21:52:51.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[85804]: from='client.54873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:51.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[85804]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-03-21T21:52:51.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[85804]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/1625650397"}]': finished 2024-03-21T21:52:51.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[85804]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T21:52:51.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[85804]: from='client.54879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:51.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:51 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2628290879' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1937719450"}]: dispatch 2024-03-21T21:52:51.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:51 smithi066 ceph-mon[68082]: from='client.54873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:51.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:51 smithi066 ceph-mon[68082]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-03-21T21:52:51.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:51 smithi066 ceph-mon[68082]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:6800/1625650397"}]': finished 2024-03-21T21:52:51.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:51 smithi066 ceph-mon[68082]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T21:52:51.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:51 smithi066 ceph-mon[68082]: from='client.54879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:51.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:51 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2628290879' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1937719450"}]: dispatch 2024-03-21T21:52:52.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:52 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2628290879' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1937719450"}]': finished 2024-03-21T21:52:52.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:52 smithi028 ceph-mon[88264]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T21:52:52.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:52 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/4198388278' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3979654249"}]: dispatch 2024-03-21T21:52:52.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:52 smithi028 ceph-mon[88264]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3979654249"}]: dispatch 2024-03-21T21:52:52.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:52 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2628290879' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1937719450"}]': finished 2024-03-21T21:52:52.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:52 smithi028 ceph-mon[85804]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T21:52:52.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:52 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/4198388278' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3979654249"}]: dispatch 2024-03-21T21:52:52.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:52 smithi028 ceph-mon[85804]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3979654249"}]: dispatch 2024-03-21T21:52:52.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:52 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2628290879' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/1937719450"}]': finished 2024-03-21T21:52:52.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:52 smithi066 ceph-mon[68082]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T21:52:52.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:52 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/4198388278' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3979654249"}]: dispatch 2024-03-21T21:52:52.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:52 smithi066 ceph-mon[68082]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3979654249"}]: dispatch 2024-03-21T21:52:53.146 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:52:53.146 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (10m) 70s ago 33m 27.7M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:52:53.146 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (10m) 70s ago 32m 83.1M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:52:53.146 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (10m) 70s ago 32m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:52:53.146 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (11m) 70s ago 37m 454M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (22m) 70s ago 39m 515M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (8m) 70s ago 39m 58.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (9m) 70s ago 37m 51.7M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (8m) 70s ago 38m 46.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (11m) 70s ago 33m 14.9M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (11m) 70s ago 33m 14.6M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (6m) 70s ago 37m 54.5M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (6m) 70s ago 36m 52.3M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (5m) 70s ago 36m 55.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (4m) 70s ago 36m 61.7M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (3m) 70s ago 35m 54.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ecd87bfd3243 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (3m) 70s ago 35m 48.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 08e1c8673055 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (2m) 70s ago 34m 54.5M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 fbe9e137d81c 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (2m) 70s ago 34m 47.2M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ededc79acc52 2024-03-21T21:52:53.147 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (11m) 70s ago 33m 71.5M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:52:53.148 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (80s) 70s ago 32m 75.4M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 08cdd63331d3 2024-03-21T21:52:53.148 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (74s) 70s ago 32m 75.2M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 b65365b5be02 2024-03-21T21:52:53.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[85804]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:52:53.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[85804]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3979654249"}]': finished 2024-03-21T21:52:53.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[85804]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T21:52:53.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/1384754810' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3107918207"}]: dispatch 2024-03-21T21:52:53.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[85804]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3107918207"}]: dispatch 2024-03-21T21:52:53.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[88264]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:52:53.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[88264]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3979654249"}]': finished 2024-03-21T21:52:53.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[88264]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T21:52:53.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/1384754810' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3107918207"}]: dispatch 2024-03-21T21:52:53.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:53 smithi028 ceph-mon[88264]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3107918207"}]: dispatch 2024-03-21T21:52:53.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:53 smithi066 ceph-mon[68082]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T21:52:53.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:53 smithi066 ceph-mon[68082]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3979654249"}]': finished 2024-03-21T21:52:53.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:53 smithi066 ceph-mon[68082]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T21:52:53.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:53 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/1384754810' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3107918207"}]: dispatch 2024-03-21T21:52:53.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:53 smithi066 ceph-mon[68082]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3107918207"}]: dispatch 2024-03-21T21:52:53.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions' 2024-03-21T21:52:54.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:54 smithi028 ceph-mon[88264]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3107918207"}]': finished 2024-03-21T21:52:54.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:54 smithi028 ceph-mon[88264]: osdmap e160: 8 total, 8 up, 8 in 2024-03-21T21:52:54.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:54 smithi028 ceph-mon[88264]: from='client.64846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:54.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:54 smithi028 ceph-mon[85804]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3107918207"}]': finished 2024-03-21T21:52:54.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:54 smithi028 ceph-mon[85804]: osdmap e160: 8 total, 8 up, 8 in 2024-03-21T21:52:54.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:54 smithi028 ceph-mon[85804]: from='client.64846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:54.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:54 smithi066 ceph-mon[68082]: from='client.? ' entity='client.iscsi.foo.smithi028.lrtsfq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.28:0/3107918207"}]': finished 2024-03-21T21:52:54.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:54 smithi066 ceph-mon[68082]: osdmap e160: 8 total, 8 up, 8 in 2024-03-21T21:52:54.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:54 smithi066 ceph-mon[68082]: from='client.64846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:52:55.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:55 smithi028 ceph-mon[85804]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:52:55.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:55 smithi028 ceph-mon[88264]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:52:55.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:55 smithi066 ceph-mon[68082]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-03-21T21:52:55.924 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:52:55.924 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:52:55.924 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:52:55.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:52:55.924 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:52:55.924 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:52:55.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:52:55.924 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:52:55.924 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 8 2024-03-21T21:52:55.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:52:55.925 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:52:55.925 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:52:55.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:52:55.925 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:52:55.925 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 15 2024-03-21T21:52:55.925 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:52:55.925 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:52:56.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:56 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2682175420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:56.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:56 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2682175420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:56.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:56 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2682175420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:52:56.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-21T21:52:57.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:57 smithi028 ceph-mon[88264]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-21T21:52:57.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:57 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:57.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:57 smithi028 ceph-mon[85804]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-21T21:52:57.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:57 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:57.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:57 smithi066 ceph-mon[68082]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-21T21:52:57.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:57 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:52:58.285 INFO:teuthology.orchestra.run.smithi028.stdout:wait for servicemap items w/ changing names to refresh 2024-03-21T21:52:58.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'sleep 60' 2024-03-21T21:52:59.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:52:59 smithi028 ceph-mon[88264]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-03-21T21:52:59.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:52:59 smithi028 ceph-mon[85804]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-03-21T21:52:59.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:52:59 smithi066 ceph-mon[68082]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-03-21T21:53:01.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:01 smithi028 ceph-mon[85804]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:01.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:53:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:53:00] "GET /metrics HTTP/1.1" 200 38477 "" "Prometheus/2.43.0" 2024-03-21T21:53:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:01 smithi028 ceph-mon[88264]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:01.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:01 smithi066 ceph-mon[68082]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:02.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:02.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:02.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:03.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:03 smithi028 ceph-mon[88264]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:53:03.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:03 smithi028 ceph-mon[85804]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:53:03.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:03 smithi066 ceph-mon[68082]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T21:53:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:05 smithi028 ceph-mon[85804]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-03-21T21:53:05.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:05 smithi028 ceph-mon[88264]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-03-21T21:53:05.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:05 smithi066 ceph-mon[68082]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-03-21T21:53:07.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:07 smithi028 ceph-mon[85804]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:07.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:07 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:07.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:07 smithi028 ceph-mon[88264]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:07.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:07 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:07.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:07 smithi066 ceph-mon[68082]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:07.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:07 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:09.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:09 smithi028 ceph-mon[85804]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:53:09.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:09 smithi028 ceph-mon[88264]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:53:09.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:09 smithi066 ceph-mon[68082]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:53:11.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:11 smithi028 ceph-mon[85804]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:11.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:11 smithi028 ceph-mon[88264]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:11.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:53:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:53:10] "GET /metrics HTTP/1.1" 200 38476 "" "Prometheus/2.43.0" 2024-03-21T21:53:11.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:11 smithi066 ceph-mon[68082]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:13.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:13 smithi028 ceph-mon[85804]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:53:13.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:13 smithi028 ceph-mon[88264]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:53:13.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:13 smithi066 ceph-mon[68082]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T21:53:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:15 smithi028 ceph-mon[85804]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:15.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:15 smithi028 ceph-mon[88264]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:15.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:15 smithi066 ceph-mon[68082]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:17.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:17 smithi028 ceph-mon[88264]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:17.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:17.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:17 smithi028 ceph-mon[85804]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:17.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:17.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:17 smithi066 ceph-mon[68082]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:17.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:18.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:18 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:18.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:18 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:18.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:18 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:19.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:19 smithi028 ceph-mon[85804]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:19.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:19 smithi028 ceph-mon[88264]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:19.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:19 smithi066 ceph-mon[68082]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:21.082 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:53:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:53:20] "GET /metrics HTTP/1.1" 200 38472 "" "Prometheus/2.43.0" 2024-03-21T21:53:21.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:21 smithi028 ceph-mon[85804]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:21.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:21 smithi028 ceph-mon[88264]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:21.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:21 smithi066 ceph-mon[68082]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:23.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:23 smithi028 ceph-mon[85804]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:23.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:23 smithi028 ceph-mon[88264]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:24.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:23 smithi066 ceph-mon[68082]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:25.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:25 smithi028 ceph-mon[85804]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:25.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:25 smithi028 ceph-mon[88264]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:26.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:25 smithi066 ceph-mon[68082]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:27.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:27 smithi028 ceph-mon[88264]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:27.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:27 smithi028 ceph-mon[85804]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:28.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:27 smithi066 ceph-mon[68082]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:28.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:28 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:28.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:28 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:29.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:28 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:29.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:29 smithi028 ceph-mon[85804]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:29.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:29 smithi028 ceph-mon[88264]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:30.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:29 smithi066 ceph-mon[68082]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:31.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:53:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:53:30] "GET /metrics HTTP/1.1" 200 38472 "" "Prometheus/2.43.0" 2024-03-21T21:53:31.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:31 smithi028 ceph-mon[85804]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:31.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:31 smithi028 ceph-mon[88264]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:32.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:31 smithi066 ceph-mon[68082]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:32.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:32 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:32.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:32 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:33.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:32 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:33.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:33 smithi028 ceph-mon[88264]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:33.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:33 smithi028 ceph-mon[85804]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:34.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:33 smithi066 ceph-mon[68082]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:35.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:35 smithi028 ceph-mon[88264]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:35.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:35 smithi028 ceph-mon[85804]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:36.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:35 smithi066 ceph-mon[68082]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:36 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:53:36.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:36 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:53:37.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:36 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T21:53:37.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:37 smithi028 ceph-mon[88264]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:37.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:37 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:53:37.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:37 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:53:37.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:37 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:53:37.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:37 smithi028 ceph-mon[85804]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:37.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:37 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:53:37.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:37 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:53:37.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:37 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:53:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:37 smithi066 ceph-mon[68082]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:37 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T21:53:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:37 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T21:53:38.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:37 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' 2024-03-21T21:53:38.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:38 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:38.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:38 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:39.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:38 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:39.901 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:39 smithi066 ceph-mon[68082]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:39.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:39 smithi028 ceph-mon[85804]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:39.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:39 smithi028 ceph-mon[88264]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:41.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:53:40 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:53:40] "GET /metrics HTTP/1.1" 200 38473 "" "Prometheus/2.43.0" 2024-03-21T21:53:41.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:41 smithi028 ceph-mon[85804]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:41.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:41 smithi028 ceph-mon[88264]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:42.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:41 smithi066 ceph-mon[68082]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:43.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:43 smithi028 ceph-mon[85804]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:43.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:43 smithi028 ceph-mon[88264]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:44.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:43 smithi066 ceph-mon[68082]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:45.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:45 smithi028 ceph-mon[85804]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:45.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:45 smithi028 ceph-mon[88264]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:46.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:45 smithi066 ceph-mon[68082]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:47.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:47 smithi028 ceph-mon[88264]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:47.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:47 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:47.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:47 smithi028 ceph-mon[85804]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:47.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:47 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:48.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:47 smithi066 ceph-mon[68082]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:48.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:47 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:53:48.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:48 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:48.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:48 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:48.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:48 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:49.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:49 smithi028 ceph-mon[85804]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:49.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:49 smithi028 ceph-mon[88264]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:50.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:49 smithi066 ceph-mon[68082]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:51.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:53:50 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:53:50] "GET /metrics HTTP/1.1" 200 38475 "" "Prometheus/2.43.0" 2024-03-21T21:53:51.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:51 smithi028 ceph-mon[85804]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:51.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:51 smithi028 ceph-mon[88264]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:52.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:51 smithi066 ceph-mon[68082]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:53.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:53 smithi028 ceph-mon[85804]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:53.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:53 smithi028 ceph-mon[88264]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:54.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:53 smithi066 ceph-mon[68082]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:55.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:55 smithi028 ceph-mon[85804]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:55.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:55 smithi028 ceph-mon[88264]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:56.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:55 smithi066 ceph-mon[68082]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:57.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:57 smithi028 ceph-mon[85804]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:57.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:57 smithi028 ceph-mon[88264]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:58.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:57 smithi066 ceph-mon[68082]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:53:58.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:58 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:58.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:58 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:59.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:58 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:53:59.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:53:59 smithi028 ceph-mon[85804]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:53:59.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:53:59 smithi028 ceph-mon[88264]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:00.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:53:59 smithi066 ceph-mon[68082]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:01.196 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:00 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:54:00] "GET /metrics HTTP/1.1" 200 38475 "" "Prometheus/2.43.0" 2024-03-21T21:54:01.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ps' 2024-03-21T21:54:01.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:01 smithi028 ceph-mon[88264]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:01.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:01 smithi028 ceph-mon[85804]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:02.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:01 smithi066 ceph-mon[68082]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:02.897 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:02 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:54:02.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:02 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:54:03.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:02 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:54:03.360 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.a smithi028 *:9093,9094 running (12m) 2m ago 34m 27.7M - 0.25.0 c8568f914cd2 a09458e22230 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.a smithi066 *:3000 running (11m) 2m ago 33m 83.1M - 9.4.12 2bacad6d85d8 c4f0c2d28ad9 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:iscsi.foo.smithi028.lrtsfq smithi028 running (11m) 2m ago 33m 58.2M - 3.5 e1d6a67b021e 59fb696f2293 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi066 *:8443,9283,8765 running (13m) 2m ago 38m 454M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 94e44685fc2d 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi028 *:8443,9283,8765 running (23m) 2m ago 40m 515M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 675d4e55bc32 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (9m) 2m ago 40m 58.6M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 66c02b062f9e 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi066 running (10m) 2m ago 39m 51.7M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ad8492014537 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi028 running (9m) 2m ago 39m 46.8M 2048M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d1de564deab3 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.a smithi028 *:9100 running (12m) 2m ago 34m 14.9M - 1.5.0 0da6a335fe13 9ceebe17c0c7 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.b smithi066 *:9100 running (12m) 2m ago 34m 14.6M - 1.5.0 0da6a335fe13 93ce3d181153 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 2m ago 38m 54.5M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 e118d84fcd79 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (7m) 2m ago 38m 52.3M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 63310efb44c3 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (6m) 2m ago 37m 55.9M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 6ff4f4ed72aa 2024-03-21T21:54:03.361 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (5m) 2m ago 37m 61.7M 2993M 19.0.0-2286-g5d87ad5b 1c3557fbb169 d5d9fb92993d 2024-03-21T21:54:03.362 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi066 running (5m) 2m ago 36m 54.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ecd87bfd3243 2024-03-21T21:54:03.362 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi066 running (4m) 2m ago 36m 48.1M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 08e1c8673055 2024-03-21T21:54:03.362 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi066 running (3m) 2m ago 36m 54.5M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 fbe9e137d81c 2024-03-21T21:54:03.362 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi066 running (3m) 2m ago 35m 47.2M 3248M 19.0.0-2286-g5d87ad5b 1c3557fbb169 ededc79acc52 2024-03-21T21:54:03.362 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.a smithi066 *:9095 running (12m) 2m ago 35m 71.5M - 2.43.0 a07b618ecd1d f8d0d1c0130b 2024-03-21T21:54:03.362 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi028.qubqiy smithi028 *:8000 running (2m) 2m ago 33m 75.4M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 08cdd63331d3 2024-03-21T21:54:03.362 INFO:teuthology.orchestra.run.smithi028.stdout:rgw.foo.smithi066.gkaqro smithi066 *:8000 running (2m) 2m ago 33m 75.2M - 19.0.0-2286-g5d87ad5b 1c3557fbb169 b65365b5be02 2024-03-21T21:54:03.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:03 smithi028 ceph-mon[88264]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:03.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:03 smithi028 ceph-mon[85804]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:03.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:03 smithi066 ceph-mon[68082]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:04.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions' 2024-03-21T21:54:04.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:04 smithi028 ceph-mon[85804]: from='client.54915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:04.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:04 smithi028 ceph-mon[88264]: from='client.54915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:05.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:04 smithi066 ceph-mon[68082]: from='client.54915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:05.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:05 smithi028 ceph-mon[85804]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:05.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:05 smithi028 ceph-mon[88264]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:06.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:05 smithi066 ceph-mon[68082]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:06.243 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:54:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-03-21T21:54:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 3 2024-03-21T21:54:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:54:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-03-21T21:54:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:54:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:54:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-03-21T21:54:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 8 2024-03-21T21:54:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:54:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": { 2024-03-21T21:54:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 2 2024-03-21T21:54:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T21:54:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-03-21T21:54:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)": 15 2024-03-21T21:54:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T21:54:06.244 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:54:06.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:06 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2374463892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:54:06.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:06 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2374463892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:54:06.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch upgrade status' 2024-03-21T21:54:06.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:06 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2374463892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:54:08.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:07 smithi066 ceph-mon[68082]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:08.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:07 smithi028 ceph-mon[88264]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:08.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:07 smithi028 ceph-mon[85804]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:08.884 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d", 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "osd", 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw", 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "15/21 daemons upgraded", 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed.", 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": true 2024-03-21T21:54:08.885 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:54:09.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:08 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:54:09.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:08 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:54:09.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:08 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:54:09.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph health detail' 2024-03-21T21:54:09.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:09 smithi028 ceph-mon[85804]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:09.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:09 smithi028 ceph-mon[88264]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:09.999 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:09 smithi066 ceph-mon[68082]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:10.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:10 smithi028 ceph-mon[88264]: from='client.54927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:10.735 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:10 smithi028 ceph-mon[85804]: from='client.54927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:10.986 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:10 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:54:10] "GET /metrics HTTP/1.1" 200 38479 "" "Prometheus/2.43.0" 2024-03-21T21:54:11.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:10 smithi066 ceph-mon[68082]: from='client.54927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:11.599 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed. 2024-03-21T21:54:11.599 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi028.lrtsfq on host smithi028 failed. 2024-03-21T21:54:11.599 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrade daemon: iscsi.foo.smithi028.lrtsfq: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi028.lrtsfq ... 2024-03-21T21:54:11.599 INFO:teuthology.orchestra.run.smithi028.stdout:Creating ceph-iscsi config... 2024-03-21T21:54:11.599 INFO:teuthology.orchestra.run.smithi028.stdout:Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/iscsi-gateway.cfg 2024-03-21T21:54:11.599 INFO:teuthology.orchestra.run.smithi028.stdout:Write file: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/iscsi.foo.smithi028.lrtsfq/tcmu-runner-entrypoint.sh 2024-03-21T21:54:11.599 INFO:teuthology.orchestra.run.smithi028.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2dc10cc158\x2de7c7\x2d11ee\x2d95cd\x2d87774f69a715.slice/ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service 2024-03-21T21:54:11.599 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl start ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout:Traceback (most recent call last): 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: return _run_code(code, main_globals, None, 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: exec(code, run_globals) 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5608, in 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 5596, in main 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3096, in command_deploy_from 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3131, in _common_deploy 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1087, in deploy_daemon 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/__main__.py", line 1223, in deploy_daemon_units 2024-03-21T21:54:11.600 INFO:teuthology.orchestra.run.smithi028.stdout: File "/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/cephadm.ec08d7de8d2fd940e5b719eeed3af2dd8dd5cbd0423d72eb1d748204cd46886f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T21:54:11.601 INFO:teuthology.orchestra.run.smithi028.stdout:RuntimeError: Failed command: systemctl start ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq: Job for ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service failed because the control process exited with error code. 2024-03-21T21:54:11.601 INFO:teuthology.orchestra.run.smithi028.stdout:See "systemctl status ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" and "journalctl -xeu ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@iscsi.foo.smithi028.lrtsfq.service" for details. 2024-03-21T21:54:11.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:11 smithi028 ceph-mon[88264]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:11.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:11 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/649748271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:54:11.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:11 smithi028 ceph-mon[85804]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:11.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:11 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/649748271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:54:12.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:11 smithi066 ceph-mon[68082]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:12.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:11 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/649748271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T21:54:12.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-21T21:54:12.863 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:12 smithi028 ceph-mon[88264]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:12.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:12 smithi028 ceph-mon[85804]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:13.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:12 smithi066 ceph-mon[68082]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:14.236 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-03-21T21:54:14.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-21T21:54:15.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:15 smithi028 ceph-mon[88264]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:15.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:15 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/2934045919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:54:15.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:15 smithi028 ceph-mon[85804]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:15.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:15 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/2934045919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:54:15.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:15 smithi066 ceph-mon[68082]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:15.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:15 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/2934045919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:54:17.016 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-2286-g5d87ad5b (5d87ad5bd031c000b3eb3ea886a68e118ae09d38) squid (dev)" 2024-03-21T21:54:17.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:17 smithi028 ceph-mon[85804]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:17.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:17 smithi028 ceph-mon[85804]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:54:17.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:17 smithi028 ceph-mon[85804]: from='client.? 172.21.15.28:0/1121859071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:54:17.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:17 smithi028 ceph-mon[88264]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:17.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:17 smithi028 ceph-mon[88264]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:54:17.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:17 smithi028 ceph-mon[88264]: from='client.? 172.21.15.28:0/1121859071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:54:17.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:17 smithi066 ceph-mon[68082]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:17.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:17 smithi066 ceph-mon[68082]: from='mgr.54108 172.21.15.28:0/2366163748' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T21:54:17.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:17 smithi066 ceph-mon[68082]: from='client.? 172.21.15.28:0/1121859071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T21:54:17.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -e sha1=5d87ad5bd031c000b3eb3ea886a68e118ae09d38 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-21T21:54:18.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:18 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:54:18.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:18 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:54:18.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:18 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:54:19.300 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:19 smithi028 ceph-mon[88264]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:19.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:19 smithi028 ceph-mon[85804]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:19.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:19 smithi066 ceph-mon[68082]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:19.740 INFO:teuthology.orchestra.run.smithi028.stdout:osd 8 2m ago - 2024-03-21T21:54:20.313 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-21T21:54:20.322 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T21:54:20.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-03-21T21:54:20.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:20 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:54:20] "GET /metrics HTTP/1.1" 200 38474 "" "Prometheus/2.43.0" 2024-03-21T21:54:21.320 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:21 smithi028 ceph-mon[88264]: from='client.54951 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:21.320 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:21 smithi028 ceph-mon[88264]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:21.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:21 smithi028 ceph-mon[85804]: from='client.54951 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:21.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:21 smithi028 ceph-mon[85804]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:21.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:21 smithi066 ceph-mon[68082]: from='client.54951 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:21.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:21 smithi066 ceph-mon[68082]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:23.156 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T21:54:23.156 INFO:teuthology.orchestra.run.smithi028.stdout: "image": "quay.io/ceph/ceph", 2024-03-21T21:54:23.156 INFO:teuthology.orchestra.run.smithi028.stdout: "registry": "quay.io", 2024-03-21T21:54:23.156 INFO:teuthology.orchestra.run.smithi028.stdout: "bare_image": "ceph/ceph", 2024-03-21T21:54:23.156 INFO:teuthology.orchestra.run.smithi028.stdout: "versions": [] 2024-03-21T21:54:23.156 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T21:54:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:23 smithi028 ceph-mon[88264]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:23 smithi028 ceph-mon[88264]: from='client.54957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:23.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:23 smithi028 ceph-mon[85804]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:23.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:23 smithi028 ceph-mon[85804]: from='client.54957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:23.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:23 smithi066 ceph-mon[68082]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:23.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:23 smithi066 ceph-mon[68082]: from='client.54957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:23.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-03-21T21:54:25.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:25 smithi028 ceph-mon[88264]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:25.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:25 smithi028 ceph-mon[85804]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:25.501 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:25 smithi066 ceph-mon[68082]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:26.606 INFO:teuthology.orchestra.run.smithi028.stdout: "16.2.0", 2024-03-21T21:54:27.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:27 smithi028 ceph-mon[85804]: from='client.64906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:27.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:27 smithi028 ceph-mon[85804]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:27.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:27 smithi028 ceph-mon[88264]: from='client.64906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:27.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:27 smithi028 ceph-mon[88264]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:27.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-03-21T21:54:27.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:27 smithi066 ceph-mon[68082]: from='client.64906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:27.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:27 smithi066 ceph-mon[68082]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T21:54:28.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:28 smithi028 ceph-mon[85804]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:54:28.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:28 smithi028 ceph-mon[88264]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:54:28.500 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:28 smithi066 ceph-mon[68082]: from='client.64786 -' entity='client.iscsi.foo.smithi028.lrtsfq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T21:54:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:29 smithi028 ceph-mon[88264]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:29.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:29 smithi028 ceph-mon[85804]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:29.499 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:29 smithi066 ceph-mon[68082]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T21:54:29.868 INFO:teuthology.orchestra.run.smithi028.stdout: "v16.2.2", 2024-03-21T21:54:29.869 INFO:teuthology.orchestra.run.smithi028.stdout: "v16.2.2-20210505", 2024-03-21T21:54:30.181 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-mon[88264]: from='client.54969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:30.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-mon[85804]: from='client.54969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:30.409 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-21T21:54:30.419 INFO:tasks.cephadm:Teardown begin 2024-03-21T21:54:30.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T21:54:30.452 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T21:54:30.486 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:30 smithi066 ceph-mon[68082]: from='client.54969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T21:54:30.492 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-21T21:54:30.492 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T21:54:30.509 DEBUG:teuthology.orchestra.run.smithi066:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T21:54:30.550 INFO:tasks.cephadm:Stopping all daemons... 2024-03-21T21:54:30.550 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-21T21:54:30.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.a 2024-03-21T21:54:30.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:30 smithi028 systemd[1]: Stopping Ceph mon.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:54:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a[85800]: 2024-03-21T21:54:30.866+0000 7f5879c04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a[85800]: 2024-03-21T21:54:30.866+0000 7f5879c04700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:31 smithi028 podman[129743]: 2024-03-21 21:54:31.030917448 +0000 UTC m=+0.278496313 container died 66c02b062f9ea5faeb7d0c05ca86383330b9fc3ff9559f3a18d86a2a8edd2c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: ::ffff:172.21.15.66 - - [21/Mar/2024:21:54:30] "GET /metrics HTTP/1.1" 200 38474 "" "Prometheus/2.43.0" 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:30] ENGINE Bus STOPPING 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:30] ENGINE Bus STOPPED 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:30] ENGINE Bus STARTING 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: CherryPy Checker: 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: The Application mounted at '' has an empty config. 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:54:31.197 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:30] ENGINE Serving on http://:::9283 2024-03-21T21:54:31.198 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:30 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:30] ENGINE Bus STARTED 2024-03-21T21:54:31.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:31 smithi028 podman[129743]: 2024-03-21 21:54:31.425241715 +0000 UTC m=+0.672820574 container cleanup 66c02b062f9ea5faeb7d0c05ca86383330b9fc3ff9559f3a18d86a2a8edd2c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-21T21:54:31.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 21 21:54:31 smithi028 bash[129743]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-a 2024-03-21T21:54:31.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.a.service' 2024-03-21T21:54:31.874 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:54:31.874 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-21T21:54:31.875 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-21T21:54:31.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.c 2024-03-21T21:54:32.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:31 smithi028 systemd[1]: Stopping Ceph mon.c for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:54:32.446 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:32] ENGINE Bus STOPPING 2024-03-21T21:54:32.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T21:54:32.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:32] ENGINE Bus STOPPED 2024-03-21T21:54:32.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:32] ENGINE Bus STARTING 2024-03-21T21:54:32.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:32] ENGINE Serving on http://:::9283 2024-03-21T21:54:32.447 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: [21/Mar/2024:21:54:32] ENGINE Bus STARTED 2024-03-21T21:54:32.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c[88240]: 2024-03-21T21:54:32.187+0000 7fc76a999700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:54:32.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:32 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c[88240]: 2024-03-21T21:54:32.187+0000 7fc76a999700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-03-21T21:54:32.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:32 smithi028 podman[129875]: 2024-03-21 21:54:32.310885055 +0000 UTC m=+0.250161974 container died d1de564deab3d7ce463fc18342ed228fe9a002e768012caa3816433980466a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-21T21:54:32.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:32 smithi028 podman[129875]: 2024-03-21 21:54:32.688752774 +0000 UTC m=+0.628029743 container cleanup d1de564deab3d7ce463fc18342ed228fe9a002e768012caa3816433980466a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:54:32.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 21 21:54:32 smithi028 bash[129875]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-c 2024-03-21T21:54:33.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.c.service' 2024-03-21T21:54:33.224 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:54:33.224 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-21T21:54:33.225 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-21T21:54:33.225 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.b 2024-03-21T21:54:33.561 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:33 smithi066 systemd[1]: Stopping Ceph mon.b for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:54:34.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b[68078]: 2024-03-21T21:54:33.559+0000 7f4caa4cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:54:34.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b[68078]: 2024-03-21T21:54:33.559+0000 7f4caa4cb700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-03-21T21:54:34.000 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:33 smithi066 podman[102466]: 2024-03-21 21:54:33.806139395 +0000 UTC m=+0.375098255 container died ad84920145372514f1edb5021ce037c84a6bc5872da1b7b14e19cd76954d9dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-21T21:54:34.471 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:34 smithi066 podman[102466]: 2024-03-21 21:54:34.227899038 +0000 UTC m=+0.796857940 container cleanup ad84920145372514f1edb5021ce037c84a6bc5872da1b7b14e19cd76954d9dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T21:54:34.472 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:34 smithi066 bash[102466]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b 2024-03-21T21:54:34.715 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.b.service' 2024-03-21T21:54:34.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:34 smithi066 podman[102480]: 2024-03-21 21:54:34.471577064 +0000 UTC m=+0.661055420 container remove ad84920145372514f1edb5021ce037c84a6bc5872da1b7b14e19cd76954d9dd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mon-b, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-21T21:54:34.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:34 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-21T21:54:34.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:34 smithi066 systemd[1]: Stopped Ceph mon.b for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:54:34.743 INFO:journalctl@ceph.mon.b.smithi066.stdout:Mar 21 21:54:34 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mon.b.service: Consumed 8.011s CPU time. 2024-03-21T21:54:34.773 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:54:34.773 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-21T21:54:34.773 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-21T21:54:34.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.y 2024-03-21T21:54:35.069 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:34 smithi028 systemd[1]: Stopping Ceph mgr.y for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:54:35.394 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:54:35.066+0000 7f1625437700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:54:35.394 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:35 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y[62474]: 2024-03-21T21:54:35.066+0000 7f1625437700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-21T21:54:35.394 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:35 smithi028 podman[130019]: 2024-03-21 21:54:35.098791702 +0000 UTC m=+0.134249779 container died 675d4e55bc325e278e88ae24f9e45c32a098bd9829dda310f092518434231616 (image=quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-21T21:54:35.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:35 smithi028 podman[130019]: 2024-03-21 21:54:35.48448753 +0000 UTC m=+0.519945606 container cleanup 675d4e55bc325e278e88ae24f9e45c32a098bd9829dda310f092518434231616 (image=quay.ceph.io/ceph-ci/ceph:5d87ad5bd031c000b3eb3ea886a68e118ae09d38, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T21:54:35.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 21 21:54:35 smithi028 bash[130019]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-y 2024-03-21T21:54:35.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-21T21:54:35.900 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:54:35.900 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-21T21:54:35.900 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-21T21:54:35.900 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x 2024-03-21T21:54:36.238 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:54:35 smithi066 systemd[1]: Stopping Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:54:36.499 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:54:36 smithi066 podman[102587]: 2024-03-21 21:54:36.238627592 +0000 UTC m=+0.145919536 container died 94e44685fc2d91678a0ec90263d2fb08c30592495f55ac3ae0e3b880f13ba8d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-21T21:54:36.859 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:54:36 smithi066 podman[102587]: 2024-03-21 21:54:36.674702775 +0000 UTC m=+0.581994725 container cleanup 94e44685fc2d91678a0ec90263d2fb08c30592495f55ac3ae0e3b880f13ba8d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-21T21:54:36.859 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:54:36 smithi066 bash[102587]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x 2024-03-21T21:54:37.098 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-21T21:54:37.126 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:54:36 smithi066 podman[102602]: 2024-03-21 21:54:36.859320091 +0000 UTC m=+0.622195604 container remove 94e44685fc2d91678a0ec90263d2fb08c30592495f55ac3ae0e3b880f13ba8d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-21T21:54:37.126 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:54:36 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:54:37.126 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:54:37 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-21T21:54:37.126 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:54:37 smithi066 systemd[1]: Stopped Ceph mgr.x for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:54:37.126 INFO:journalctl@ceph.mgr.x.smithi066.stdout:Mar 21 21:54:37 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@mgr.x.service: Consumed 24.096s CPU time. 2024-03-21T21:54:37.142 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:54:37.142 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-21T21:54:37.142 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-21T21:54:37.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.0 2024-03-21T21:54:37.446 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:37 smithi028 systemd[1]: Stopping Ceph osd.0 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:54:37.946 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:37 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[95744]: 2024-03-21T21:54:37.455+0000 7f10b203e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:54:37.946 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:37 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[95744]: 2024-03-21T21:54:37.455+0000 7f10b203e700 -1 osd.0 160 *** Got signal Terminated *** 2024-03-21T21:54:37.946 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:37 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0[95744]: 2024-03-21T21:54:37.455+0000 7f10b203e700 -1 osd.0 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:54:41.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:54:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:54:41.411Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-03-21T21:54:41.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:54:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:54:41.411Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-03-21T21:54:41.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:54:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:54:41.411Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-03-21T21:54:41.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:54:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:54:41.412Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-03-21T21:54:41.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:54:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:54:41.412Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-03-21T21:54:43.149 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:42 smithi028 podman[130141]: 2024-03-21 21:54:42.8468432 +0000 UTC m=+5.508372602 container died e118d84fcd79b9dc04b014ffd2186f3a855b1dcc82e345f6029358f743cd22da (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T21:54:43.446 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:43 smithi028 podman[130141]: 2024-03-21 21:54:43.23940679 +0000 UTC m=+5.900936173 container cleanup e118d84fcd79b9dc04b014ffd2186f3a855b1dcc82e345f6029358f743cd22da (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-21T21:54:43.446 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:43 smithi028 bash[130141]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0 2024-03-21T21:54:43.834 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:43 smithi028 podman[130158]: 2024-03-21 21:54:43.499004556 +0000 UTC m=+0.652061415 container remove e118d84fcd79b9dc04b014ffd2186f3a855b1dcc82e345f6029358f743cd22da (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-21T21:54:44.103 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:43 smithi028 podman[130250]: 2024-03-21 21:54:43.765475524 +0000 UTC m=+0.018607763 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:54:44.103 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:43 smithi028 podman[130250]: 2024-03-21 21:54:43.891416782 +0000 UTC m=+0.144549007 container create ab3427120576e26ed147024bde862b3e71ffeb8a3ae1602a4395fe834d13df54 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T21:54:44.411 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:44 smithi028 podman[130250]: 2024-03-21 21:54:44.102712336 +0000 UTC m=+0.355844568 container init ab3427120576e26ed147024bde862b3e71ffeb8a3ae1602a4395fe834d13df54 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-21T21:54:44.411 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:44 smithi028 podman[130250]: 2024-03-21 21:54:44.106881663 +0000 UTC m=+0.360013885 container start ab3427120576e26ed147024bde862b3e71ffeb8a3ae1602a4395fe834d13df54 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-21T21:54:44.411 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:44 smithi028 podman[130250]: 2024-03-21 21:54:44.169130658 +0000 UTC m=+0.422262899 container attach ab3427120576e26ed147024bde862b3e71ffeb8a3ae1602a4395fe834d13df54 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-21T21:54:44.696 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:44 smithi028 podman[130250]: 2024-03-21 21:54:44.411451173 +0000 UTC m=+0.664583397 container died ab3427120576e26ed147024bde862b3e71ffeb8a3ae1602a4395fe834d13df54 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-21T21:54:45.018 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 21 21:54:44 smithi028 podman[130277]: 2024-03-21 21:54:44.997946657 +0000 UTC m=+0.581056146 container remove ab3427120576e26ed147024bde862b3e71ffeb8a3ae1602a4395fe834d13df54 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:54:45.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.0.service' 2024-03-21T21:54:45.067 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:54:45.068 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-21T21:54:45.068 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-21T21:54:45.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.1 2024-03-21T21:54:45.292 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:45 smithi028 systemd[1]: Stopping Ceph osd.1 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:54:45.696 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[100536]: 2024-03-21T21:54:45.392+0000 7fd6b507f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:54:45.696 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[100536]: 2024-03-21T21:54:45.392+0000 7fd6b507f700 -1 osd.1 160 *** Got signal Terminated *** 2024-03-21T21:54:45.696 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1[100536]: 2024-03-21T21:54:45.392+0000 7fd6b507f700 -1 osd.1 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:54:51.077 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:50 smithi028 podman[130359]: 2024-03-21 21:54:50.767836154 +0000 UTC m=+5.487602200 container died 63310efb44c3a339ebc3ddb92bba5787ec9f2dd5526e14fb48d909b8d1b9f4f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T21:54:51.429 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:51 smithi028 podman[130359]: 2024-03-21 21:54:51.17594579 +0000 UTC m=+5.895711838 container cleanup 63310efb44c3a339ebc3ddb92bba5787ec9f2dd5526e14fb48d909b8d1b9f4f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-21T21:54:51.430 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:51 smithi028 bash[130359]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1 2024-03-21T21:54:51.696 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:51 smithi028 podman[130387]: 2024-03-21 21:54:51.427153865 +0000 UTC m=+0.655309755 container remove 63310efb44c3a339ebc3ddb92bba5787ec9f2dd5526e14fb48d909b8d1b9f4f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0) 2024-03-21T21:54:52.038 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:51 smithi028 podman[130480]: 2024-03-21 21:54:51.652597405 +0000 UTC m=+0.017765115 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:54:52.039 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:51 smithi028 podman[130480]: 2024-03-21 21:54:51.76986644 +0000 UTC m=+0.135034177 container create e082c2ad3b73b831046156b4b4fecb4167d857cde7cb8fa87a0a758ba50c4c66 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T21:54:52.039 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:51 smithi028 podman[130480]: 2024-03-21 21:54:51.964355256 +0000 UTC m=+0.329522944 container init e082c2ad3b73b831046156b4b4fecb4167d857cde7cb8fa87a0a758ba50c4c66 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-21T21:54:52.039 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:51 smithi028 podman[130480]: 2024-03-21 21:54:51.968738678 +0000 UTC m=+0.333906372 container start e082c2ad3b73b831046156b4b4fecb4167d857cde7cb8fa87a0a758ba50c4c66 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-21T21:54:52.039 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:52 smithi028 podman[130480]: 2024-03-21 21:54:52.038829753 +0000 UTC m=+0.403997465 container attach e082c2ad3b73b831046156b4b4fecb4167d857cde7cb8fa87a0a758ba50c4c66 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-21T21:54:52.446 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 21 21:54:52 smithi028 podman[130480]: 2024-03-21 21:54:52.264113488 +0000 UTC m=+0.629281175 container died e082c2ad3b73b831046156b4b4fecb4167d857cde7cb8fa87a0a758ba50c4c66 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-21T21:54:52.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.1.service' 2024-03-21T21:54:52.928 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:54:52.928 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-21T21:54:52.928 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-21T21:54:52.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.2 2024-03-21T21:54:53.196 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:53 smithi028 systemd[1]: Stopping Ceph osd.2 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:54:53.553 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[107169]: 2024-03-21T21:54:53.278+0000 7f0ac8d36700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:54:53.554 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[107169]: 2024-03-21T21:54:53.278+0000 7f0ac8d36700 -1 osd.2 160 *** Got signal Terminated *** 2024-03-21T21:54:53.554 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2[107169]: 2024-03-21T21:54:53.278+0000 7f0ac8d36700 -1 osd.2 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:54:53.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:54:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:54:53.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-21T21:54:53.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:54:53 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:54:53.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:54:59.071 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:58 smithi028 podman[130584]: 2024-03-21 21:54:58.762796982 +0000 UTC m=+5.607355187 container died 6ff4f4ed72aa03a9f58e4917ced558ce01016c2f23feb2ff9828d52ed19027a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-21T21:54:59.379 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:59 smithi028 podman[130584]: 2024-03-21 21:54:59.17023924 +0000 UTC m=+6.014797487 container cleanup 6ff4f4ed72aa03a9f58e4917ced558ce01016c2f23feb2ff9828d52ed19027a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-21T21:54:59.379 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:59 smithi028 bash[130584]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2 2024-03-21T21:54:59.697 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:59 smithi028 podman[130601]: 2024-03-21 21:54:59.379416637 +0000 UTC m=+0.619002350 container remove 6ff4f4ed72aa03a9f58e4917ced558ce01016c2f23feb2ff9828d52ed19027a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-21T21:55:00.107 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:59 smithi028 podman[130696]: 2024-03-21 21:54:59.704804682 +0000 UTC m=+0.018059450 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:55:00.107 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:54:59 smithi028 podman[130696]: 2024-03-21 21:54:59.813517038 +0000 UTC m=+0.126771801 container create 976902f3b03f5acfe691f255cdd7149b8e458e85eb5a0d785665d0c8e8609127 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-21T21:55:00.107 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:55:00 smithi028 podman[130696]: 2024-03-21 21:55:00.032755583 +0000 UTC m=+0.346010348 container init 976902f3b03f5acfe691f255cdd7149b8e458e85eb5a0d785665d0c8e8609127 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T21:55:00.107 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:55:00 smithi028 podman[130696]: 2024-03-21 21:55:00.035874489 +0000 UTC m=+0.349129262 container start 976902f3b03f5acfe691f255cdd7149b8e458e85eb5a0d785665d0c8e8609127 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T21:55:00.108 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:55:00 smithi028 podman[130696]: 2024-03-21 21:55:00.107480274 +0000 UTC m=+0.420735041 container attach 976902f3b03f5acfe691f255cdd7149b8e458e85eb5a0d785665d0c8e8609127 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T21:55:00.446 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:55:00 smithi028 podman[130696]: 2024-03-21 21:55:00.337601656 +0000 UTC m=+0.650856421 container died 976902f3b03f5acfe691f255cdd7149b8e458e85eb5a0d785665d0c8e8609127 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2) 2024-03-21T21:55:00.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.2.service' 2024-03-21T21:55:00.875 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:55:00 smithi028 podman[130722]: 2024-03-21 21:55:00.827712971 +0000 UTC m=+0.479882662 container remove 976902f3b03f5acfe691f255cdd7149b8e458e85eb5a0d785665d0c8e8609127 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-21T21:55:00.875 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:55:00 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-21T21:55:00.876 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:55:00 smithi028 systemd[1]: Stopped Ceph osd.2 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:55:00.876 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 21 21:55:00 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.2.service: Consumed 5.162s CPU time. 2024-03-21T21:55:00.890 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:55:00.891 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-21T21:55:00.891 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-21T21:55:00.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.3 2024-03-21T21:55:01.196 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:00 smithi028 systemd[1]: Stopping Ceph osd.3 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:55:01.696 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:01 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[113574]: 2024-03-21T21:55:01.231+0000 7f1cd6127700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:55:01.696 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:01 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[113574]: 2024-03-21T21:55:01.231+0000 7f1cd6127700 -1 osd.3 160 *** Got signal Terminated *** 2024-03-21T21:55:01.696 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:01 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3[113574]: 2024-03-21T21:55:01.231+0000 7f1cd6127700 -1 osd.3 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:55:03.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:03.404+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:03.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:03.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:55:03.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:03.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-21T21:55:03.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:03 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:03.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:55:04.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:03 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:03.758+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:04.748 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:04.427+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:04.748 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:04 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:04.747+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:05.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:05.435+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:06.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:05 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:05.784+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:06.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:06 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:06.354+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:06.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:06 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:06.434+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:06.832 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:06 smithi028 podman[130801]: 2024-03-21 21:55:06.557352296 +0000 UTC m=+5.444594041 container died d5d9fb92993d4190b94978299f8f9d52197ae9fcee8c1d4f383bda2f0d305063 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:55:07.181 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:06 smithi028 podman[130801]: 2024-03-21 21:55:06.930214372 +0000 UTC m=+5.817456117 container cleanup d5d9fb92993d4190b94978299f8f9d52197ae9fcee8c1d4f383bda2f0d305063 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-21T21:55:07.181 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:06 smithi028 bash[130801]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3 2024-03-21T21:55:07.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:06 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:06.777+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:07.446 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:07 smithi028 podman[130830]: 2024-03-21 21:55:07.181087758 +0000 UTC m=+0.623925081 container remove d5d9fb92993d4190b94978299f8f9d52197ae9fcee8c1d4f383bda2f0d305063 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:55:07.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:07.402+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:07.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:07.410+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:07.767 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:07 smithi028 podman[130920]: 2024-03-21 21:55:07.490284888 +0000 UTC m=+0.108506086 container create eb0ea4aaa72dd2700e6d085f7f2a3bc3f6f1526e49f52351f17e1a3a846fcfff (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-21T21:55:07.767 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:07 smithi028 podman[130920]: 2024-03-21 21:55:07.399844925 +0000 UTC m=+0.018066122 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:55:07.768 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:07 smithi028 podman[130920]: 2024-03-21 21:55:07.70103977 +0000 UTC m=+0.319260976 container init eb0ea4aaa72dd2700e6d085f7f2a3bc3f6f1526e49f52351f17e1a3a846fcfff (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-21T21:55:07.768 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:07 smithi028 podman[130920]: 2024-03-21 21:55:07.704171349 +0000 UTC m=+0.322392548 container start eb0ea4aaa72dd2700e6d085f7f2a3bc3f6f1526e49f52351f17e1a3a846fcfff (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:55:07.768 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:07 smithi028 podman[130920]: 2024-03-21 21:55:07.76772293 +0000 UTC m=+0.385944137 container attach eb0ea4aaa72dd2700e6d085f7f2a3bc3f6f1526e49f52351f17e1a3a846fcfff (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:55:08.196 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:08 smithi028 podman[130920]: 2024-03-21 21:55:08.007000559 +0000 UTC m=+0.625221749 container died eb0ea4aaa72dd2700e6d085f7f2a3bc3f6f1526e49f52351f17e1a3a846fcfff (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T21:55:08.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:07.794+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:08.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:07 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:07.904+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:08.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.3.service' 2024-03-21T21:55:08.559 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:08 smithi028 podman[130946]: 2024-03-21 21:55:08.512467802 +0000 UTC m=+0.499359488 container remove eb0ea4aaa72dd2700e6d085f7f2a3bc3f6f1526e49f52351f17e1a3a846fcfff (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-03-21T21:55:08.560 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:08 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-21T21:55:08.560 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:08 smithi028 systemd[1]: Stopped Ceph osd.3 for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:55:08.560 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 21 21:55:08 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.3.service: Consumed 6.437s CPU time. 2024-03-21T21:55:08.576 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:55:08.576 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-21T21:55:08.576 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-21T21:55:08.577 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.4 2024-03-21T21:55:08.613 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:08.361+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:08.614 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:08.375+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:08.903 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:08 smithi066 systemd[1]: Stopping Ceph osd.4 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:55:08.903 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:08.804+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:09.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:08.902+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:09.250 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:08.911+0000 7ff63270b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:55:09.250 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:08.911+0000 7ff63270b700 -1 osd.4 160 *** Got signal Terminated *** 2024-03-21T21:55:09.250 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:08 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:08.911+0000 7ff63270b700 -1 osd.4 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:55:09.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:09.340+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:09.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:09.403+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:10.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:09.850+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:10.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:09 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:09.916+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:10.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:10.379+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:10.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:10.291+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:11.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:10.848+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:11.251 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:10 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:10.933+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:11.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:11.271+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:11.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:11.405+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:12.231 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:11.822+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:12.231 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:11.909+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:12.231 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:11 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:11.909+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:12.500 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:12.230+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:12.500 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:12.424+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:13.249 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:12.861+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:13.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:12.954+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:13.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:12.954+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:13.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4[78409]: 2024-03-21T21:55:13.270+0000 7ff62ecff700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:36.588369+0000 front 2024-03-21T21:54:36.588478+0000 (oldest deadline 2024-03-21T21:55:02.988814+0000) 2024-03-21T21:55:13.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:13.404+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:13.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:13.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:55:13.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:13.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-21T21:55:13.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:13 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:13.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:55:14.244 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:13.895+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:14.245 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:13.895+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.681349+0000 front 2024-03-21T21:54:47.681414+0000 (oldest deadline 2024-03-21T21:55:13.581201+0000) 2024-03-21T21:55:14.245 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:13.977+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:14.245 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:13 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:13.977+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:14.500 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:14 smithi066 podman[102724]: 2024-03-21 21:55:14.244628799 +0000 UTC m=+5.456524653 container died ecd87bfd3243118f47e47e0b4552b185d1ff747be10eeeba0f289f33d10b6ce7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-21T21:55:14.500 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:14.434+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:15.071 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:14 smithi066 podman[102724]: 2024-03-21 21:55:14.803504298 +0000 UTC m=+6.015400152 container cleanup ecd87bfd3243118f47e47e0b4552b185d1ff747be10eeeba0f289f33d10b6ce7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:55:15.071 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:14 smithi066 bash[102724]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4 2024-03-21T21:55:15.072 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:14.903+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:15.072 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:14.903+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.681349+0000 front 2024-03-21T21:54:47.681414+0000 (oldest deadline 2024-03-21T21:55:13.581201+0000) 2024-03-21T21:55:15.072 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:14.963+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:15.072 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:14 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:14.963+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:15.429 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:15 smithi066 podman[102753]: 2024-03-21 21:55:15.071606365 +0000 UTC m=+0.824637088 container remove ecd87bfd3243118f47e47e0b4552b185d1ff747be10eeeba0f289f33d10b6ce7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-21T21:55:15.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:15.479+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:15.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:15 smithi066 podman[102848]: 2024-03-21 21:55:15.332261532 +0000 UTC m=+0.019063987 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:55:15.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:15 smithi066 podman[102848]: 2024-03-21 21:55:15.439196031 +0000 UTC m=+0.125998514 container create 686bdae64a4b6db99cfd4501c9856344e93245cf5b87fe9efb114c7e1393db0d (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:55:15.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:15 smithi066 podman[102848]: 2024-03-21 21:55:15.641949499 +0000 UTC m=+0.328751952 container init 686bdae64a4b6db99cfd4501c9856344e93245cf5b87fe9efb114c7e1393db0d (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-21T21:55:15.750 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:15 smithi066 podman[102848]: 2024-03-21 21:55:15.645331467 +0000 UTC m=+0.332133910 container start 686bdae64a4b6db99cfd4501c9856344e93245cf5b87fe9efb114c7e1393db0d (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T21:55:16.013 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:15.913+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:16.013 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:15 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:15.913+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.681349+0000 front 2024-03-21T21:54:47.681414+0000 (oldest deadline 2024-03-21T21:55:13.581201+0000) 2024-03-21T21:55:16.013 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:15 smithi066 podman[102848]: 2024-03-21 21:55:15.750463958 +0000 UTC m=+0.437266417 container attach 686bdae64a4b6db99cfd4501c9856344e93245cf5b87fe9efb114c7e1393db0d (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-21T21:55:16.013 INFO:journalctl@ceph.osd.4.smithi066.stdout:Mar 21 21:55:15 smithi066 podman[102848]: 2024-03-21 21:55:15.942389872 +0000 UTC m=+0.629192314 container died 686bdae64a4b6db99cfd4501c9856344e93245cf5b87fe9efb114c7e1393db0d (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-21T21:55:16.295 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:16.012+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:16.296 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:16.012+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:16.604 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:16.503+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:16.604 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:16.503+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:16.624 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.4.service' 2024-03-21T21:55:16.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:55:16.713 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-21T21:55:16.713 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-21T21:55:16.713 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.5 2024-03-21T21:55:16.909 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:16 smithi066 systemd[1]: Stopping Ceph osd.5 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:55:16.909 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:16.908+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:17.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:16.971+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:17.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:16.971+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:17.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:16 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:16.908+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.681349+0000 front 2024-03-21T21:54:47.681414+0000 (oldest deadline 2024-03-21T21:55:13.581201+0000) 2024-03-21T21:55:17.251 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:17.124+0000 7fee212d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:55:17.251 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:17.124+0000 7fee212d3700 -1 osd.5 160 *** Got signal Terminated *** 2024-03-21T21:55:17.251 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:17.124+0000 7fee212d3700 -1 osd.5 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:55:17.749 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:17.483+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:17.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:17.483+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:18.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:17.948+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:18.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:17 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:17.948+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.681349+0000 front 2024-03-21T21:54:47.681414+0000 (oldest deadline 2024-03-21T21:55:13.581201+0000) 2024-03-21T21:55:18.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:18.020+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:18.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:18.020+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:18.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:18.436+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:18.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:18.436+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:19.249 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:18.904+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:19.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:18 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:18.904+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.681349+0000 front 2024-03-21T21:54:47.681414+0000 (oldest deadline 2024-03-21T21:55:13.581201+0000) 2024-03-21T21:55:19.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:19.067+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:19.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:19.067+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:19.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:19.456+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:19.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:19.456+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:20.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:19.869+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:20.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:19 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:19.869+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.681349+0000 front 2024-03-21T21:54:47.681414+0000 (oldest deadline 2024-03-21T21:55:13.581201+0000) 2024-03-21T21:55:20.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:20.029+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:20.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:20.029+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:20.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:20.452+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:20.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:20.452+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:21.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:21.042+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:21.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:21.042+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:21.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:20.851+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:21.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:20.851+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.681349+0000 front 2024-03-21T21:54:47.681414+0000 (oldest deadline 2024-03-21T21:55:13.581201+0000) 2024-03-21T21:55:21.251 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:20 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:20.851+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:54.682333+0000 front 2024-03-21T21:54:54.682229+0000 (oldest deadline 2024-03-21T21:55:19.982246+0000) 2024-03-21T21:55:21.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:21.405+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:21.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:21.405+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:22.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:21.892+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:40.179516+0000 front 2024-03-21T21:54:40.179499+0000 (oldest deadline 2024-03-21T21:55:03.079477+0000) 2024-03-21T21:55:22.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:21.892+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.681349+0000 front 2024-03-21T21:54:47.681414+0000 (oldest deadline 2024-03-21T21:55:13.581201+0000) 2024-03-21T21:55:22.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5[83395]: 2024-03-21T21:55:21.892+0000 7fee1d8c7700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:54.682333+0000 front 2024-03-21T21:54:54.682229+0000 (oldest deadline 2024-03-21T21:55:19.982246+0000) 2024-03-21T21:55:22.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:21.994+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:22.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:21.994+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:22.251 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:21 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:21.994+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:22.701 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:22 smithi066 podman[102955]: 2024-03-21 21:55:22.451953309 +0000 UTC m=+5.544804195 container died 08e1c86730554d8a8ed25fb9809e5b1ae6365aad19885d1d671a440a65cb2834 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-21T21:55:22.702 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:22.378+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:22.702 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:22.378+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:22.966 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:22 smithi066 podman[102955]: 2024-03-21 21:55:22.808189459 +0000 UTC m=+5.901040335 container cleanup 08e1c86730554d8a8ed25fb9809e5b1ae6365aad19885d1d671a440a65cb2834 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_BRANCH=HEAD) 2024-03-21T21:55:22.967 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:22 smithi066 bash[102955]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5 2024-03-21T21:55:23.250 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:23 smithi066 podman[102974]: 2024-03-21 21:55:23.075878625 +0000 UTC m=+0.627107769 container remove 08e1c86730554d8a8ed25fb9809e5b1ae6365aad19885d1d671a440a65cb2834 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-21T21:55:23.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:22.965+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:23.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:22.965+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:23.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:22 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:22.965+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:23.617 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:23.357+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:23.618 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:23.357+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:23.618 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:23 smithi066 podman[103068]: 2024-03-21 21:55:23.334866098 +0000 UTC m=+0.018701482 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:55:23.618 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:23 smithi066 podman[103068]: 2024-03-21 21:55:23.460236895 +0000 UTC m=+0.144072271 container create ef98252cfb53df6c62acd42af8a5dca428d222fd403403670ae49c46e75da2e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:55:23.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:23.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:55:23.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:23.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-21T21:55:23.946 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:23 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:23.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:55:24.072 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:23 smithi066 podman[103068]: 2024-03-21 21:55:23.75460821 +0000 UTC m=+0.438443599 container init ef98252cfb53df6c62acd42af8a5dca428d222fd403403670ae49c46e75da2e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T21:55:24.073 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:23 smithi066 podman[103068]: 2024-03-21 21:55:23.757823446 +0000 UTC m=+0.441658824 container start ef98252cfb53df6c62acd42af8a5dca428d222fd403403670ae49c46e75da2e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:55:24.073 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:23 smithi066 podman[103068]: 2024-03-21 21:55:23.83791908 +0000 UTC m=+0.521754457 container attach ef98252cfb53df6c62acd42af8a5dca428d222fd403403670ae49c46e75da2e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-21T21:55:24.073 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:23.968+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:24.073 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:23.968+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:24.073 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:23 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:23.968+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:24.337 INFO:journalctl@ceph.osd.5.smithi066.stdout:Mar 21 21:55:24 smithi066 podman[103068]: 2024-03-21 21:55:24.073258567 +0000 UTC m=+0.757093943 container died ef98252cfb53df6c62acd42af8a5dca428d222fd403403670ae49c46e75da2e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-21T21:55:24.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:24.336+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:24.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:24.336+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:24.795 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.5.service' 2024-03-21T21:55:24.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:55:24.845 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-21T21:55:24.845 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-21T21:55:24.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.6 2024-03-21T21:55:25.146 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:24 smithi066 systemd[1]: Stopping Ceph osd.6 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:55:25.146 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:24.961+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:25.146 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:24.961+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:25.146 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:24 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:24.961+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:25.500 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:25.144+0000 7fb200cca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:55:25.500 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:25.144+0000 7fb200cca700 -1 osd.6 160 *** Got signal Terminated *** 2024-03-21T21:55:25.500 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:25.144+0000 7fb200cca700 -1 osd.6 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:55:25.500 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:25.379+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:25.500 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:25.379+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:25.500 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:25 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:25.379+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:57.438637+0000 front 2024-03-21T21:54:57.438613+0000 (oldest deadline 2024-03-21T21:55:24.438824+0000) 2024-03-21T21:55:26.380 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:26.001+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:26.380 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:26.001+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:26.380 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:26.001+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:26.749 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:26.379+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:26.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:26.379+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:26.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:26.379+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:57.438637+0000 front 2024-03-21T21:54:57.438613+0000 (oldest deadline 2024-03-21T21:55:24.438824+0000) 2024-03-21T21:55:27.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:26.963+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:27.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:26.963+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:27.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:26 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:26.963+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:27.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:27 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:27.370+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:27.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:27 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:27.370+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:27.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:27 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:27.370+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:57.438637+0000 front 2024-03-21T21:54:57.438613+0000 (oldest deadline 2024-03-21T21:55:24.438824+0000) 2024-03-21T21:55:28.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:27 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:27.922+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:28.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:27 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:27.922+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:28.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:27 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:27.922+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:28.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:28.405+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:28.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:28.405+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:28.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:28.405+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:57.438637+0000 front 2024-03-21T21:54:57.438613+0000 (oldest deadline 2024-03-21T21:55:24.438824+0000) 2024-03-21T21:55:29.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:28.881+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:29.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:28.881+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:29.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:28.881+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:29.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:28 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:28.881+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:29.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:29 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:29.412+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:41.036460+0000 front 2024-03-21T21:54:41.036487+0000 (oldest deadline 2024-03-21T21:55:06.336441+0000) 2024-03-21T21:55:29.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:29 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:29.412+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:46.337041+0000 front 2024-03-21T21:54:46.337002+0000 (oldest deadline 2024-03-21T21:55:16.337548+0000) 2024-03-21T21:55:29.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:29 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:29.412+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:57.438637+0000 front 2024-03-21T21:54:57.438613+0000 (oldest deadline 2024-03-21T21:55:24.438824+0000) 2024-03-21T21:55:29.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:29 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6[88388]: 2024-03-21T21:55:29.412+0000 7fb1f92b6700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.439256+0000 front 2024-03-21T21:55:04.439195+0000 (oldest deadline 2024-03-21T21:55:28.539256+0000) 2024-03-21T21:55:30.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:29 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:29.906+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:30.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:29 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:29.906+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:30.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:29 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:29.906+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:30.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:29 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:29.906+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:30.750 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:30 smithi066 podman[103173]: 2024-03-21 21:55:30.498869812 +0000 UTC m=+5.464109485 container died fbe9e137d81ceb0c250de4b01dec7ad3d06e51f43710581ba08c31134419df0a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-21T21:55:31.163 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:30 smithi066 podman[103173]: 2024-03-21 21:55:30.828378046 +0000 UTC m=+5.793617727 container cleanup fbe9e137d81ceb0c250de4b01dec7ad3d06e51f43710581ba08c31134419df0a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-21T21:55:31.163 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:30 smithi066 bash[103173]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6 2024-03-21T21:55:31.163 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:30.864+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:31.163 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:30.864+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:31.163 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:30.864+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:31.163 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:30 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:30.864+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:31.499 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:31 smithi066 podman[103190]: 2024-03-21 21:55:31.163180053 +0000 UTC m=+0.658474294 container remove fbe9e137d81ceb0c250de4b01dec7ad3d06e51f43710581ba08c31134419df0a (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T21:55:32.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:31.816+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:32.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:31.816+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:32.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:31.816+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:32.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:31 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:31.816+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:32.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:31 smithi066 podman[103284]: 2024-03-21 21:55:31.511636561 +0000 UTC m=+0.019523139 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:55:32.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:31 smithi066 podman[103284]: 2024-03-21 21:55:31.62268841 +0000 UTC m=+0.130574971 container create c4c21f3a2dfab06fd9d3e91570c2ffe3f3d5b00f41cba7124c01eaf6ddfbde40 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5) 2024-03-21T21:55:32.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:31 smithi066 podman[103284]: 2024-03-21 21:55:31.825119648 +0000 UTC m=+0.333006219 container init c4c21f3a2dfab06fd9d3e91570c2ffe3f3d5b00f41cba7124c01eaf6ddfbde40 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-21T21:55:32.001 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:31 smithi066 podman[103284]: 2024-03-21 21:55:31.831163287 +0000 UTC m=+0.339049848 container start c4c21f3a2dfab06fd9d3e91570c2ffe3f3d5b00f41cba7124c01eaf6ddfbde40 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:55:32.446 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:32 smithi066 podman[103284]: 2024-03-21 21:55:32.108790675 +0000 UTC m=+0.616677237 container attach c4c21f3a2dfab06fd9d3e91570c2ffe3f3d5b00f41cba7124c01eaf6ddfbde40 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-21T21:55:32.446 INFO:journalctl@ceph.osd.6.smithi066.stdout:Mar 21 21:55:32 smithi066 podman[103284]: 2024-03-21 21:55:32.137317517 +0000 UTC m=+0.645204079 container died c4c21f3a2dfab06fd9d3e91570c2ffe3f3d5b00f41cba7124c01eaf6ddfbde40 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-21T21:55:32.799 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.6.service' 2024-03-21T21:55:32.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:55:32.849 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-21T21:55:32.850 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-21T21:55:32.850 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.7 2024-03-21T21:55:33.141 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:32.810+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:33.142 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:32.810+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:33.142 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:32.810+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:33.142 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:32 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:32.810+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:33.142 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:32 smithi066 systemd[1]: Stopping Ceph osd.7 for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:55:33.499 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:33.140+0000 7fed40653700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T21:55:33.500 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:33.140+0000 7fed40653700 -1 osd.7 160 *** Got signal Terminated *** 2024-03-21T21:55:33.500 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:33.140+0000 7fed40653700 -1 osd.7 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T21:55:33.857 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:33.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-21T21:55:33.857 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:33.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-21T21:55:33.857 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:33 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:33.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:55:34.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:33.803+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:34.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:33.803+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:34.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:33.803+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:34.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:33 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:33.803+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:35.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:34 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:34.830+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:35.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:34 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:34.830+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:35.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:34 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:34.830+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:35.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:34 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:34.830+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:35.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:34 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:34.830+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.66:6804 osd.4 since back 2024-03-21T21:55:11.716646+0000 front 2024-03-21T21:55:11.716745+0000 (oldest deadline 2024-03-21T21:55:34.616672+0000) 2024-03-21T21:55:36.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:35.867+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:36.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:35.867+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:36.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:35.867+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:36.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:35.867+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:36.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:35 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:35.867+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.66:6804 osd.4 since back 2024-03-21T21:55:11.716646+0000 front 2024-03-21T21:55:11.716745+0000 (oldest deadline 2024-03-21T21:55:34.616672+0000) 2024-03-21T21:55:37.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:36.909+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:37.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:36.909+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:37.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:36.909+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:37.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:36.909+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:37.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:36 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:36.909+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.66:6804 osd.4 since back 2024-03-21T21:55:11.716646+0000 front 2024-03-21T21:55:11.716745+0000 (oldest deadline 2024-03-21T21:55:34.616672+0000) 2024-03-21T21:55:38.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:37.902+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-21T21:54:42.012595+0000 front 2024-03-21T21:54:42.012708+0000 (oldest deadline 2024-03-21T21:55:07.312616+0000) 2024-03-21T21:55:38.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:37.902+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-03-21T21:54:47.313178+0000 front 2024-03-21T21:54:47.313294+0000 (oldest deadline 2024-03-21T21:55:11.413029+0000) 2024-03-21T21:55:38.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:37.902+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-03-21T21:54:55.314906+0000 front 2024-03-21T21:54:55.314847+0000 (oldest deadline 2024-03-21T21:55:21.214810+0000) 2024-03-21T21:55:38.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:37.902+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-03-21T21:55:04.715876+0000 front 2024-03-21T21:55:04.715966+0000 (oldest deadline 2024-03-21T21:55:28.815911+0000) 2024-03-21T21:55:38.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:37 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7[93384]: 2024-03-21T21:55:37.902+0000 7fed3cc47700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.66:6804 osd.4 since back 2024-03-21T21:55:11.716646+0000 front 2024-03-21T21:55:11.716745+0000 (oldest deadline 2024-03-21T21:55:34.616672+0000) 2024-03-21T21:55:38.966 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:38 smithi066 podman[103392]: 2024-03-21 21:55:38.688626244 +0000 UTC m=+5.642872134 container died ededc79acc52d19540667b0828c49ad633673b60d127d681fe9fc2da3d2dde64 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-21T21:55:39.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:39 smithi066 podman[103392]: 2024-03-21 21:55:39.174064517 +0000 UTC m=+6.128310403 container cleanup ededc79acc52d19540667b0828c49ad633673b60d127d681fe9fc2da3d2dde64 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:55:39.250 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:39 smithi066 bash[103392]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7 2024-03-21T21:55:39.742 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:39 smithi066 podman[103409]: 2024-03-21 21:55:39.425453798 +0000 UTC m=+0.734764876 container remove ededc79acc52d19540667b0828c49ad633673b60d127d681fe9fc2da3d2dde64 (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-21T21:55:40.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:39 smithi066 podman[103504]: 2024-03-21 21:55:39.742968093 +0000 UTC m=+0.114376222 container create 3e791830b29441158a3ee958ca503d578753932a555eeeccff1b50dde4d767ed (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-21T21:55:40.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:39 smithi066 podman[103504]: 2024-03-21 21:55:39.647085377 +0000 UTC m=+0.018493491 image pull 1c3557fbb169403f5161927e77b9b2186268246b87fb5a1554a7b0a8b4e49068 quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d 2024-03-21T21:55:40.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:39 smithi066 podman[103504]: 2024-03-21 21:55:39.937525962 +0000 UTC m=+0.308934076 container init 3e791830b29441158a3ee958ca503d578753932a555eeeccff1b50dde4d767ed (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T21:55:40.000 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:39 smithi066 podman[103504]: 2024-03-21 21:55:39.941000969 +0000 UTC m=+0.312409058 container start 3e791830b29441158a3ee958ca503d578753932a555eeeccff1b50dde4d767ed (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, ceph=True, GIT_CLEAN=True) 2024-03-21T21:55:40.500 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:40 smithi066 podman[103504]: 2024-03-21 21:55:40.02954379 +0000 UTC m=+0.400951884 container attach 3e791830b29441158a3ee958ca503d578753932a555eeeccff1b50dde4d767ed (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-21T21:55:40.500 INFO:journalctl@ceph.osd.7.smithi066.stdout:Mar 21 21:55:40 smithi066 podman[103504]: 2024-03-21 21:55:40.237887908 +0000 UTC m=+0.609295997 container died 3e791830b29441158a3ee958ca503d578753932a555eeeccff1b50dde4d767ed (image=quay.ceph.io/ceph-ci/ceph@sha256:f60f3c216f555b8e367335520c66ffb3591c4a024f1368149f6143364f3d390d, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1131-5d87ad5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-21T21:55:40.815 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@osd.7.service' 2024-03-21T21:55:40.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:55:40.863 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-21T21:55:40.863 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-21T21:55:40.863 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a 2024-03-21T21:55:41.168 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:40 smithi066 systemd[1]: Stopping Ceph prometheus.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.168Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.168Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.168Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.168Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.168Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.168Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.169Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.169Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.169Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-21T21:55:41.444 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.169Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T21:55:41.445 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.169Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-21T21:55:41.445 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a[64492]: ts=2024-03-21T21:55:41.169Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-21T21:55:41.445 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 podman[103608]: 2024-03-21 21:55:41.178941458 +0000 UTC m=+0.118185546 container died f8d0d1c0130ba275b128708f377dcbf51f2e9fb9339c81cae57af1029eeb7288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:55:41.749 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 podman[103608]: 2024-03-21 21:55:41.643573126 +0000 UTC m=+0.582817222 container cleanup f8d0d1c0130ba275b128708f377dcbf51f2e9fb9339c81cae57af1029eeb7288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T21:55:41.750 INFO:journalctl@ceph.prometheus.a.smithi066.stdout:Mar 21 21:55:41 smithi066 bash[103608]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T21:55:42.027 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-21T21:55:42.070 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T21:55:42.070 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-21T21:55:42.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T21:55:43.874 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:55:43 smithi028 systemd[1]: Stopping Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:55:43.875 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:43.556Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-21T21:55:43.875 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:43.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-21T21:55:43.875 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:43 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:43.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi066.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.66:8443: connect: connection refused" 2024-03-21T21:55:44.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:55:43 smithi028 podman[131315]: 2024-03-21 21:55:43.874196156 +0000 UTC m=+0.113672985 container died 9ceebe17c0c78fcccf0452f69871f43144633877ef129cab6908df869571ad2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:55:44.506 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:55:44 smithi028 podman[131315]: 2024-03-21 21:55:44.313259432 +0000 UTC m=+0.552736254 container cleanup 9ceebe17c0c78fcccf0452f69871f43144633877ef129cab6908df869571ad2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:55:44.506 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:55:44 smithi028 bash[131315]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a 2024-03-21T21:55:44.761 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:55:44 smithi028 podman[131327]: 2024-03-21 21:55:44.505929678 +0000 UTC m=+0.631951106 container remove 9ceebe17c0c78fcccf0452f69871f43144633877ef129cab6908df869571ad2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T21:55:44.761 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:55:44 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:55:44.761 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:55:44 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-21T21:55:44.761 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:55:44 smithi028 systemd[1]: Stopped Ceph node-exporter.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:55:44.761 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 21 21:55:44 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 3.160s CPU time. 2024-03-21T21:55:45.114 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:44 smithi028 systemd[1]: Stopping Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:55:45.446 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:45 smithi028 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a[79217]: ts=2024-03-21T21:55:45.242Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-21T21:55:45.446 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:45 smithi028 podman[131438]: 2024-03-21 21:55:45.283638029 +0000 UTC m=+0.214282628 container stop a09458e22230b7cbf566694f9004cc2c136baf339f6f835dfb16bed05184ea03 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:55:45.446 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:45 smithi028 podman[131438]: 2024-03-21 21:55:45.309052798 +0000 UTC m=+0.239697397 container died a09458e22230b7cbf566694f9004cc2c136baf339f6f835dfb16bed05184ea03 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:55:45.802 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:45 smithi028 podman[131438]: 2024-03-21 21:55:45.617915432 +0000 UTC m=+0.548560030 container cleanup a09458e22230b7cbf566694f9004cc2c136baf339f6f835dfb16bed05184ea03 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:55:45.802 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:45 smithi028 bash[131438]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T21:55:46.169 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:45 smithi028 podman[131452]: 2024-03-21 21:55:45.80210668 +0000 UTC m=+0.513585068 container remove a09458e22230b7cbf566694f9004cc2c136baf339f6f835dfb16bed05184ea03 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T21:55:46.169 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:45 smithi028 podman[131452]: 2024-03-21 21:55:45.886003117 +0000 UTC m=+0.597481474 volume remove 5539a45423ea6137983ee7a206cf1a7f2f8c43543067faebd8a4bda26b8cde6e 2024-03-21T21:55:46.446 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:46 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T21:55:46.446 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:46 smithi028 systemd[1]: Stopped Ceph alertmanager.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:55:46.446 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 21 21:55:46 smithi028 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.578s CPU time. 2024-03-21T21:56:09.289 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T21:56:11.026 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:56:10 smithi066 systemd[1]: Stopping Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:56:11.291 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:56:11 smithi066 podman[104017]: 2024-03-21 21:56:11.025798519 +0000 UTC m=+0.122779692 container died 93ce3d18115351dd6465735d23a3fabd52ccd870156165b0b2a272854882f37e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:56:11.649 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:56:11 smithi066 podman[104017]: 2024-03-21 21:56:11.448684035 +0000 UTC m=+0.545665247 container cleanup 93ce3d18115351dd6465735d23a3fabd52ccd870156165b0b2a272854882f37e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:56:11.650 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:56:11 smithi066 bash[104017]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b 2024-03-21T21:56:12.000 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:56:11 smithi066 podman[104031]: 2024-03-21 21:56:11.649809511 +0000 UTC m=+0.624608263 container remove 93ce3d18115351dd6465735d23a3fabd52ccd870156165b0b2a272854882f37e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T21:56:12.000 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:56:11 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-21T21:56:12.000 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:56:11 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-21T21:56:12.000 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:56:11 smithi066 systemd[1]: Stopped Ceph node-exporter.b for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:56:12.000 INFO:journalctl@ceph.node-exporter.b.smithi066.stdout:Mar 21 21:56:11 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 3.298s CPU time. 2024-03-21T21:56:12.299 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:12 smithi066 systemd[1]: Stopping Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715... 2024-03-21T21:56:12.704 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=server t=2024-03-21T21:56:12.303202612Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-21T21:56:12.704 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=ticker t=2024-03-21T21:56:12.303439338Z level=info msg=stopped last_tick=2024-03-21T21:56:10Z 2024-03-21T21:56:12.704 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:12 smithi066 ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a[66457]: logger=tracing t=2024-03-21T21:56:12.303440015Z level=info msg="Closing tracing" 2024-03-21T21:56:12.704 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:12 smithi066 podman[104138]: 2024-03-21 21:56:12.313524622 +0000 UTC m=+0.112337561 container died c4f0c2d28ad98f188e4d730ed924c87e6830db4e452846ac632b3923da8b288a (image=quay.io/ceph/grafana:9.4.12, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T21:56:12.999 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:12 smithi066 podman[104138]: 2024-03-21 21:56:12.785696012 +0000 UTC m=+0.584508967 container cleanup c4f0c2d28ad98f188e4d730ed924c87e6830db4e452846ac632b3923da8b288a (image=quay.io/ceph/grafana:9.4.12, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T21:56:13.000 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:12 smithi066 bash[104138]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a 2024-03-21T21:56:13.402 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:13 smithi066 podman[104153]: 2024-03-21 21:56:13.05367281 +0000 UTC m=+0.738665802 container remove c4f0c2d28ad98f188e4d730ed924c87e6830db4e452846ac632b3923da8b288a (image=quay.io/ceph/grafana:9.4.12, name=ceph-c10cc158-e7c7-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T21:56:13.403 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:13 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-21T21:56:13.403 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:13 smithi066 systemd[1]: Stopped Ceph grafana.a for c10cc158-e7c7-11ee-95cd-87774f69a715. 2024-03-21T21:56:13.403 INFO:journalctl@ceph.grafana.a.smithi066.stdout:Mar 21 21:56:13 smithi066 systemd[1]: ceph-c10cc158-e7c7-11ee-95cd-87774f69a715@grafana.a.service: Consumed 11.840s CPU time. 2024-03-21T21:56:24.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T21:56:24.966 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T21:56:25.000 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-21T21:56:25.002 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118/remote/smithi028/crash 2024-03-21T21:56:25.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/crash -- . 2024-03-21T21:56:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T21:56:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-03-21T21:56:25.057 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118/remote/smithi066/crash 2024-03-21T21:56:25.057 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/crash -- . 2024-03-21T21:56:25.125 INFO:teuthology.orchestra.run.smithi066.stderr:tar: /var/lib/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T21:56:25.125 INFO:teuthology.orchestra.run.smithi066.stderr:tar: Error is not recoverable: exiting now 2024-03-21T21:56:25.126 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-21T21:56:25.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-21T21:56:25.156 INFO:teuthology.orchestra.run.smithi028.stdout:2024-03-21T21:15:00.864416+0000 mon.a (mon.0) 167 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T21:56:25.158 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-21T21:56:25.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[SEC\]' /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-21T21:56:25.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]' /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-21T21:56:25.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[WRN\]' /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-21T21:56:25.367 INFO:teuthology.orchestra.run.smithi028.stdout:2024-03-21T21:15:00.864416+0000 mon.a (mon.0) 167 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T21:56:25.369 INFO:tasks.cephadm:Compressing logs... 2024-03-21T21:56:25.369 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T21:56:25.412 DEBUG:teuthology.orchestra.run.smithi066:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T21:56:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T21:56:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:: No such file or directory 2024-03-21T21:56:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-21T21:56:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T21:56:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-21T21:56:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mon.a.log: /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-21T21:56:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-21T21:56:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-21T21:56:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-21T21:56:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T21:56:25.440 INFO:teuthology.orchestra.run.smithi028.stderr: 89.9% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-21T21:56:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log 2024-03-21T21:56:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.0.log: /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.cephadm.log: 94.4% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T21:56:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-21T21:56:25.444 INFO:teuthology.orchestra.run.smithi066.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-03-21T21:56:25.444 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-21T21:56:25.444 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-21T21:56:25.444 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T21:56:25.444 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-21T21:56:25.444 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-21T21:56:25.445 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-21T21:56:25.446 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.4.log: /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi066.gkaqro.log 2024-03-21T21:56:25.447 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-21T21:56:25.447 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi066.gkaqro.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T21:56:25.447 INFO:teuthology.orchestra.run.smithi066.stderr: 76.3% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi066.gkaqro.log.gz 2024-03-21T21:56:25.449 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log: 94.4% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T21:56:25.449 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T21:56:25.450 INFO:teuthology.orchestra.run.smithi028.stderr: 93.9% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T21:56:25.450 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi028.qubqiy.log 2024-03-21T21:56:25.451 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mgr.x.log: 91.1% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T21:56:25.452 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log 2024-03-21T21:56:25.452 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T21:56:25.453 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log: 90.7% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T21:56:25.454 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-volume.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T21:56:25.454 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-21T21:56:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi028.qubqiy.log: 76.3% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi028.qubqiy.log.gz 2024-03-21T21:56:25.457 INFO:teuthology.orchestra.run.smithi066.stderr: 89.1% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T21:56:25.458 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-21T21:56:25.466 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.1% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T21:56:25.475 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T21:56:25.479 INFO:teuthology.orchestra.run.smithi066.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T21:56:25.606 INFO:teuthology.orchestra.run.smithi066.stderr: 89.7% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-21T21:56:25.964 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-21T21:56:26.021 INFO:teuthology.orchestra.run.smithi028.stderr: 92.4% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-21T21:56:26.084 INFO:teuthology.orchestra.run.smithi066.stderr: 92.5% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-21T21:56:26.866 INFO:teuthology.orchestra.run.smithi028.stderr: 93.3% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-21T21:56:27.069 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-21T21:56:27.249 INFO:teuthology.orchestra.run.smithi066.stderr: 93.1% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-21T21:56:27.328 INFO:teuthology.orchestra.run.smithi066.stderr: 93.8% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-21T21:56:27.345 INFO:teuthology.orchestra.run.smithi066.stderr: 93.3% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-21T21:56:27.405 INFO:teuthology.orchestra.run.smithi028.stderr: 93.4% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-21T21:56:27.757 INFO:teuthology.orchestra.run.smithi066.stderr: 93.5% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-21T21:56:27.759 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-21T21:56:27.759 INFO:teuthology.orchestra.run.smithi066.stderr:real 0m2.332s 2024-03-21T21:56:27.759 INFO:teuthology.orchestra.run.smithi066.stderr:user 0m8.471s 2024-03-21T21:56:27.759 INFO:teuthology.orchestra.run.smithi066.stderr:sys 0m0.305s 2024-03-21T21:56:27.809 INFO:teuthology.orchestra.run.smithi028.stderr: 93.4% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-21T21:56:28.163 INFO:teuthology.orchestra.run.smithi028.stderr: 93.4% -- replaced with /var/log/ceph/c10cc158-e7c7-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-21T21:56:28.165 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T21:56:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m2.741s 2024-03-21T21:56:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m10.885s 2024-03-21T21:56:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m0.383s 2024-03-21T21:56:28.166 INFO:tasks.cephadm:Archiving logs... 2024-03-21T21:56:28.166 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118/remote/smithi028/log 2024-03-21T21:56:28.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T21:56:29.644 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118/remote/smithi066/log 2024-03-21T21:56:29.645 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T21:56:30.924 INFO:tasks.cephadm:Removing cluster... 2024-03-21T21:56:30.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 --force 2024-03-21T21:56:31.220 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c10cc158-e7c7-11ee-95cd-87774f69a715 --force 2024-03-21T21:56:31.526 INFO:tasks.cephadm:Removing cephadm ... 2024-03-21T21:56:31.526 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T21:56:31.542 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T21:56:31.557 INFO:tasks.cephadm:Teardown complete 2024-03-21T21:56:31.557 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-21T21:56:31.567 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-21T21:56:31.567 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T21:56:31.584 DEBUG:teuthology.orchestra.run.smithi066:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T21:56:31.596 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-03-21T21:56:31.600 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T21:56:31.600 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-03-21T21:56:31.600 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 98 +1165us[+1167us] +/- 143ms 2024-03-21T21:56:31.600 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -258us[ -256us] +/- 62ms 2024-03-21T21:56:31.600 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -312us[ -310us] +/- 91ms 2024-03-21T21:56:31.600 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:56:31.611 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-21T21:56:31.615 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T21:56:31.615 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-21T21:56:31.615 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 36 +1199us[+1199us] +/- 144ms 2024-03-21T21:56:31.615 INFO:teuthology.orchestra.run.smithi066.stdout:^* hv02.front.sepia.ceph.com 3 7 377 99 -242us[ -244us] +/- 61ms 2024-03-21T21:56:31.616 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -277us[ -277us] +/- 91ms 2024-03-21T21:56:31.616 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T21:56:31.616 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-21T21:56:31.627 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-21T21:56:31.628 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-21T21:56:31.651 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T21:56:31.745 DEBUG:teuthology.orchestra.run.smithi066:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T21:56:31.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-21T21:56:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T21:56:31.866 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-21T21:56:31.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T21:56:31.901 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-21T21:56:31.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-21T21:56:31.926 INFO:teuthology.task.internal:Duration was 2897.891947 seconds 2024-03-21T21:56:31.926 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-21T21:56:31.961 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-21T21:56:31.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T21:56:31.963 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T21:56:32.003 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T21:56:32.014 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T21:56:32.487 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-21T21:56:32.487 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T21:56:32.487 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T21:56:32.548 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi066.front.sepia.ceph.com 2024-03-21T21:56:32.548 DEBUG:teuthology.orchestra.run.smithi066:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T21:56:32.579 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-21T21:56:32.579 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T21:56:32.592 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T21:56:32.653 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-21T21:56:32.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T21:56:32.662 DEBUG:teuthology.orchestra.run.smithi066:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T21:56:33.249 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-21T21:56:33.260 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-21T21:56:33.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T21:56:33.287 DEBUG:teuthology.orchestra.run.smithi066:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T21:56:33.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-21T21:56:33.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T21:56:33.331 DEBUG:teuthology.orchestra.run.smithi066:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T21:56:33.353 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-03-21T21:56:33.387 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = core 2024-03-21T21:56:33.419 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T21:56:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T21:56:33.433 DEBUG:teuthology.orchestra.run.smithi066:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T21:56:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T21:56:33.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-21T21:56:33.484 INFO:teuthology.task.internal:Transferring archived files... 2024-03-21T21:56:33.485 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118/remote/smithi028 2024-03-21T21:56:33.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T21:56:33.536 DEBUG:teuthology.misc:Transferring archived files from smithi066:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118/remote/smithi066 2024-03-21T21:56:33.537 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T21:56:33.583 INFO:teuthology.task.internal:Removing archive directory... 2024-03-21T21:56:33.583 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T21:56:33.585 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T21:56:33.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-21T21:56:33.643 INFO:teuthology.task.internal:Not uploading archives. 2024-03-21T21:56:33.644 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-21T21:56:33.669 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-21T21:56:33.669 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T21:56:33.671 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T21:56:33.683 INFO:teuthology.orchestra.run.smithi028.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 21:56 /home/ubuntu/cephtest 2024-03-21T21:56:33.686 INFO:teuthology.orchestra.run.smithi066.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 21:56 /home/ubuntu/cephtest 2024-03-21T21:56:33.687 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-21T21:56:33.703 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-21T21:56:33.769 INFO:teuthology.nuke:Checking targets against current locks 2024-03-21T21:56:33.799 DEBUG:teuthology.nuke:shortname: smithi028 2024-03-21T21:56:33.799 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T21:56:33.824 DEBUG:teuthology.nuke:shortname: smithi066 2024-03-21T21:56:33.824 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T21:56:33.828 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 20:56:36.632640', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLT+w6gzfjApPcFVQPVNGJ+GM/Hovgy+qhTKc8HqxrjO92ZOc/GXZL9GTaOmmgnH3XPucBJusQF0Xx4Ay4DuFzM='} 2024-03-21T21:56:33.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-03-21_20:44:17-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1131-distro-default-smithi/7615118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 20:56:36.631243', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEiwQxsdZiuPkLNNIeSatr9t2ofZNna2bdq/zpWeMK5Qe71r7U87/AkpS7nRo1oixXHMhrMWscMeHRk5t6o9gk='} 2024-03-21T21:56:33.883 INFO:teuthology.orchestra.console.smithi028:Power off 2024-03-21T21:56:33.884 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T21:56:33.909 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-03-21T21:56:33.910 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T21:56:33.926 INFO:teuthology.orchestra.console.smithi066:Power off 2024-03-21T21:56:33.926 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T21:56:33.938 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-03-21T21:56:33.951 DEBUG:teuthology.orchestra.console.smithi066:power off output: Chassis Power Control: Down/Off 2024-03-21T21:56:33.951 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T21:56:33.978 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-21T21:56:37.940 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T21:56:37.979 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T21:56:38.050 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-03-21T21:56:38.089 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-21T21:56:42.054 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T21:56:42.090 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T21:56:42.168 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-03-21T21:56:42.200 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off 2024-03-21T21:56:42.269 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-03-21T21:56:42.301 INFO:teuthology.orchestra.console.smithi066:Power off completed 2024-03-21T21:56:42.407 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 2897.891947031021 failure_reason: '"2024-03-21T21:15:00.864416+0000 mon.a (mon.0) 167 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' owner: scheduled_phlogistonjohn@teuthology success: false 2024-03-21T21:56:42.407 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T21:56:42.495 INFO:teuthology.run:FAIL